Skip to main content

Define a descriptive scenario

In this tutorial you will learn how to create a scenario to simulate the historic performance of the example business and produce descriptive insights, you will:

  1. Create sources for procurement, sales, and supplier data from JSON files.
  2. Use the PipelineBuilder to process and transform data streams from the sources, defining fields and output keys.
  3. Create functions to generate key dates related to sales and procurement.
  4. Define resources such as cash, liability, inventory, etc., mapping them from values or data streams.
  5. Define processes for sales, receiving goods from suppliers, paying suppliers, and procurement.
  6. Build a reporter process to generate hourly reports on cash, liability, and inventory.
  7. Create a descriptive scenario to simulate historic business outcomes, incorporating resources and processes.
  8. Build layouts for graphs displaying cash balance, inventory level, and liability over time.
  9. Construct a dashboard with tabs for different layouts, a header displaying key performance indicators.

This lesson will assume that you have an empty project and asset which you can to deploy to a workspace named 04_03_01_define_a_descriptive_scenario with the following command:

edk template deploy -ycw 04_03_01_define_a_descriptive_scenario

Data files

To build the solution for the example business, some pre-defined data can be used. You will use data provided, which can be found in the data directory in the reference project.

Procurement data

The procurement data will be in a JSONL format describing the orders over time as the date/time and the supplier name, you can see a sample below:

{"date":"2023-10-29T17:00:00.000Z","supplierName":"Josh's Sausages"}
{"date":"2023-10-30T17:00:00.000Z","supplierName":"Josh's Sausages"}

Supplier data

The supplier data will be in a JSONL format describing the individual suppliers and associated information, you can see a sample below:

{"supplierName":"Bob's Dogs","paymentTerms":1,"leadTime":1,"unitCost":2.05,"unitQty":"250"}
{"supplierName":"Josh's Sausages","paymentTerms":2,"leadTime":2,"unitCost":1.75,"unitQty":"950"}
{"supplierName":"Meat Kings","paymentTerms":1.5,"leadTime":0.125,"unitCost":2,"unitQty":"650"}

Sales data

The sales data will be in a JSONL format describing the aggregated sales per hour describing the date/time, the qty sold in that hour, and the average discount in the hour, you can see a sample below:

{"date":"2023-10-29T09:00:00.000Z","qty":"51","price":2.0900000000000003}
{"date":"2023-10-29T10:00:00.000Z","qty":"105","price":1.5620000000000003}
{"date":"2023-10-29T11:00:00.000Z","qty":"140","price":1.9800000000000002}

Constant values

At times you will undertake solutions which use known constant and static values, as an example, assume that along with the files, you were provided with the regular retail price, which will be defined as a constant typescript value in the project:

const rrp = 2.20;

Later you will use this value as the price of the products in sales.

Define data sources

To use the input files, you can create file sources using a SourceBuilder, by taking the following steps:

  1. add a SourceBuilder for each file to parse
  2. make a file source and define the path

In the code add the above changes:

const procurement_file = new SourceBuilder("Procurement File").file({ path: 'data/procurement.jsonl' })
const sales_file = new SourceBuilder("Sales File").file({ path: 'data/sales.jsonl' })
const suppliers_file = new SourceBuilder("Suppliers File").file({ path: 'data/suppliers.jsonl' })

Define parsing of blob data sources

After defining data sources, you can parse the data into a tabular format using the PipelineBuilder, by taking the following steps:

  1. add a PipelineBuilder for each file to parse
  2. add the file source output to each PipelineBuilder
  3. define the parsed type using fromJsonLines
    1. define the fields to parse and their type (matching the input data)
    2. define the output kay as a value that is unique

In the code add the above changes:

const sales_data = new PipelineBuilder('Sales')
    .from(sales_file.outputStream())
    .fromJsonLines({
        fields: {
            // the date of the aggregate sales records
            date: DateTimeType,
            // the qty sold in the hour
            qty: IntegerType,
            // the price applied during that hour
            price: FloatType,
        },
        // the sale date is unique, so can be used as the key
        output_key: fields => Print(fields.date)
    })

Repeat similar steps for supplier_data and procurement_data.

Define key date calculation

To perform simulation in a scenario, it is necessary to know when the simulation needs to being and end, you can use the sales records as a reference for date/time range. You can calculate the key dates for simulation using a FunctionBuilder, with the following steps:

  1. add a FunctionBuilder
  2. add an input of the sales data
  3. add a function body, advantage
    1. calculate the last observable data with a Reduce, starting at DefaultValue which is the unix epoch
    2. starting from the last observed date, calculate the first sale date
    3. Return the first and last dates

In the code add the above changes:

const sales_dates = new FunctionBuilder("Sales Dates")
    .input("sales", sales_data.outputStream())
    .body(builder => builder
        // when the last sale hour occurred (this is the current date time)
        .let("last", vars => Reduce(
            vars.sales,
            (prev, sale) => Max(prev, GetField(sale, 'date')),
            DefaultValue(DateTimeType))
        )
        // when the first sale hour occurred (historically)
        .let("first", vars => Reduce(
            vars.sales,
            (prev, sale) => Min(prev, GetField(sale, 'date')),
            vars.last
        ))
        // return all the dates
        .return({
            last: vars => vars.last,
            first: vars => vars.first,
        })
    )

Define resources

To perform simulation, you need to define the resources that you identified for the example business. Resources have types, and can be 'mapped' from data, or values which creates resources of the same type as the mapped data. You can define the resources using the ResourceBuilder with the following steps:

  1. add a ResourceBuilder for the Cash, Liability, Inventory and Price which are mapped (created) from constant values using mapFromValue
    1. for Cash, Liability, Inventory you can choose the actual value communicated by the business owner
    2. for the price you can choose the "average" based on the bounds of price communicated with the business owner
  2. add a ResourceBuilder for the Suppliers, which are mapped from the values in supplier_data
  3. add a ResourceBuilder for the Reports, which are hourly performance reports, starting with an empty collection

In the code add the above changes:

const cash = new ResourceBuilder("Cash").mapFromValue(500)
const liability = new ResourceBuilder("Liability").mapFromValue(0)
const inventory = new ResourceBuilder("Inventory").mapFromValue(2500n)
const price = new ResourceBuilder("Price").mapFromValue(rrp - (rrp - rrp * 0.7) / 2)
const suppliers = new ResourceBuilder("Suppliers").mapFromStream(supplier_data.outputStream())
// create a report resource to store the hourly reports
const reports = new ResourceBuilder("Reports").mapFromValue(
    new Map(),
    DictType(
        StringType,
        StructType({
            date: DateTimeType,
            cash: FloatType,
            liability: FloatType,
            inventory: IntegerType,
        })
    )
)
Mapping Resources

Note that mapping describes how resources are initially define in simulation.

  • mapFromStream: will perform a data driven mapping which will define the initial value of a resource from the current value of a data stream.
  • mapFromValue: will automatically perform data driven mapping, by creating a value source for the resource and using the value to initialise the resource in simulation
Resources Type

When mapping a resource from a value, like a value source, if the value in type script doesn't define the type, it will need to be defined manually. In the above example, the Reports resource is mapped from a new Map(). Since the type key and value isn't expressed by an empty Map, the type is manually defined.

The signature of mapFromValue is as follows, where the first argument is the value, and the second is the type of the value:

mapFromValue(value: ValueTypeOf<EastType>, type: EastType)

Define processes

Once you have created processes, you can start to create the processes that will be used to simulate the business with the ProcessBuilder.

Define sales process

The value network describes the relationships between sales, and resources, as follows:

You can define a sales process to reflect the diagram, with the following steps:

  1. add a ProcessBuilder for the Sales process
  2. add the resources that are used in the process
  3. add the values that are used in the process (which need to exist in the data set)
    1. A ProcessBuilder has an implicit date value, which is the date/time of the process
  4. add the logic to calculate the qty sold and amount of money in the hour
  5. add the logic to update the inventory and cash balance

In the code add the above changes:

// create the sales process that exchanges cash for inventory
const sales = new ProcessBuilder("Sales")
    .resource(price)
    .resource(inventory)
    .resource(cash)
    .value("qty", IntegerType)
    .value("price", FloatType)
    // can only sell if there is enough inventory, so update the  qty
    .assign("qty", (props, resources) => Min(resources["Inventory"], props.qty))
    // get the total amount of revenue
    .let("amount", props => Multiply(props.qty, props.price))
    // update the inventory balance by the qty
    .set("Inventory", (props, resources) => Subtract(resources["Inventory"], props.qty))
    // update the cash balance by the amount
    .set("Cash", (props, resources) => Add(resources.Cash, props.amount))
    // the initial data comes from the historic sale data
    .mapManyFromStream(sales_data.outputStream())

ProcessBuilder vs FunctionBuilder

Defining a ProcessBuilder is similar to a FunctionBuilder, though it has a few differences:

  • it can 'import' and use or change resources and processes
  • it can have values defined, which could be though of as function arguments
  • a process always needs to have a date, which represents when the process was undertaken
  • it can be mapped from data or values
  • it does NOT require definition of return values or a body

Other than the above, many of the methods available in a function can be used in a process, such as assign, set, let, forDict, etc.

ProcessBuilder Type

Since a process has an implicit date value, the will always be at least:

DictType(StringType, StructType({ date: DateTimeType }))

In the above example, the type of the Sales process is

DictType(StringType, StructType({ date: DateTimeType, qty: IntegerType, price: FloatType }))

Define procurement process

The procurement process is more complex than sales, since it involves performing other processes at later dates. The value network describes the relationships between procurement, and resources, as follows:

You can define the procurement process to reflect the diagram by creating the different processes as per the value network.

Define receiving goods process

First, you can define the process to receive goods from a supplier, with the following steps:

  1. add a ProcessBuilder for the Receive Goods process
  2. add the inventory resource to reference and/or in the process
  3. add the qty value as the qty of inventory to receive
    1. remember a ProcessBuilder has an implicit date value, which is the date/time of the process
  4. add the logic to update the inventory by the qty

In the code add the above changes:

// receive stock from a supplier, place in inventory
const receive_goods = new ProcessBuilder("Receive Goods")
    .resource(inventory)
    .value("qty", IntegerType)
    // update the inventory by the qty
    .set("Inventory", (props, resources) => Add(resources.Inventory, props.qty))

Define supplier payment process

Next, you can define the process to pay a supplier, with the following steps:

  1. add a ProcessBuilder for the Pay Supplier process
  2. add the cash, liability, and inventory resources to reference and/or set in the process
  3. add the supplierName, unitCost, and qty values as the supplier name, unit cost, and qty of inventory to pay for
    1. remember a ProcessBuilder has an implicit date value, which is the date/time of the process
  4. add the logic to calculate the total amount to pay the supplier
  5. add the logic to update the liability and cash by the amount

In the code add the above changes:

// pay the supplier for some ordered inventory, and clear the liability
const pay_supplier = new ProcessBuilder("Pay Supplier")
    .resource(cash)
    .resource(liability)
    .resource(inventory)
    .value("supplierName", StringType)
    .value("unitCost", FloatType)
    .value("qty", IntegerType)
    // the total amount to be paid
    .let("amount", props => Multiply(props.qty, props.unitCost))
    // the debt has been cleared
    .set("Liability", (props, resources) => Add(resources.Liability, props.amount))
    // update the cash by the amount
    .set("Cash", (props, resources) => Subtract(resources.Cash, props.amount))

Define procurement process

Lastly you can complete the procurement process, with the following steps:

  1. add a ProcessBuilder for the Procurement process
  2. add the cash, liability, inventory, and suppliers resources to reference and/or set in the process
  3. add the other procurement processes as steps in the process
  4. add the supplierName value as the supplier name
    1. remember a ProcessBuilder has an implicit date value, which is the date/time of the process
  5. add the logic to get the supplier since the Supplier resource is a collection
  6. add the logic to calculate the total amount to pay the supplier
  7. add a conditional block to only order if there is enough cash available
    1. update the Liability by the amount
    2. execute the Receive Goods process, passing the calculated date, supplier name, and qty
    3. execute the Pay Supplier process, passing the calculated date, supplier name, unit cost, and qty
  8. add the logic to map the initial data from the historic procurement data using mapManyFromStream

In the code add the above changes:

// order some inventory, schedule supplier payment and receiving the inventory later 
const procurement = new ProcessBuilder("Procurement")
    .resource(cash)
    .resource(suppliers)
    .process(pay_supplier)
    .process(receive_goods)
    .resource(liability)
    .value("supplierName", StringType)
    // get the supplier
    .let("supplier", (props, resources) => Get(resources.Suppliers, props.supplierName))
    // calculate the total amount to pay the supplier
    .let("amount", (props) => Multiply(GetField(props.supplier, "unitCost"), GetField(props.supplier, "unitQty")))
    // only order if there is enough cash available
    .if(
        (props, resources) => GreaterEqual(resources.Cash, props.amount),
        block => block
            .set("Liability", (props, resources) => Subtract(resources.Liability, props.amount))
            // schedule receiving the inventory
            .execute("Receive Goods", props => Struct({
                date: AddDuration(props.date, GetField(props.supplier, "leadTime"), "day"),
                supplierName: props.supplierName,
                qty: GetField(props.supplier, "unitQty")
            }))
            // schedule paying the supplier
            .execute("Pay Supplier", props => Struct({
                date: AddDuration(props.date, GetField(props.supplier, "paymentTerms"), "day"),
                supplierName: props.supplierName,
                unitCost: GetField(props.supplier, "unitCost"),
                qty: GetField(props.supplier, "unitQty"),
            })),
    )
    // the initial data comes from the historic procurement data
    .mapManyFromStream(procurement_data.outputStream())

Process Execution

A helpful way to view process execution, is that it is a way to schedule other processes to be executed at a later date. In the above example, the Procurement process schedules the Receive Goods and Pay Supplier processes to be executed at a later date.

Define reporting

You have now defined the equivalent of the value network for the example business, and can now define a process to generate hourly reports. Creating a process for reporting allows us to replicate something that often happens in a business, which is observing and recording the values of important resources over time.

Use ProcessBuilder to build a process that creates hourly reports for cash, liability, and inventory with the following steps:

  1. add a ProcessBuilder for the Reporter process
  2. add the cash, liability, inventory, and reports resources to reference in the process
  3. add the logic to insert a report into the reports resource
    1. the key of the report is the date
    2. the value of the report is a struct containing the date, cash, liability, and inventory
  4. add the logic to create another report in an hour - by executing the Reporter process
  5. add the logic to map

In the code add the above changes:

// create the hourly reports
const reporter = new ProcessBuilder("Reporter")
    .resource(cash)
    .resource(inventory)
    .resource(reports)
    .resource(liability)
    // insert a report into the reports resource
    .insert(
        (_props, resources) => resources.Reports,
        (props, _resources) => Print(props.date),
        (props, resources) => Struct({
            date: props.date,
            cash: resources.Cash,
            liability: resources.Liability,
            inventory: resources.Inventory,
        })
    )
    // create another report in an hour
    .execute("Reporter", props => Struct({ date: AddDuration(props.date, 1, "hour") }))
    // the first report should start at the first sale date (in the past)
    .mapFromPipeline(builder => builder
        .from(sales_dates.outputStreams().first)
        .transform((date) => Struct({ date }))
    )

Process Recursion

Like a function in javascript, a process can call itself. In the above example, the Reporter process calls itself to create another report in an hour. The ability for recursion results in a need for similar care and testing that would be applied in javascript programming, and because of this, it is recommended to define an end date as shown in the following section.

Elara does attempt to automatically detect and prevent infinite recursion, and cancel accidental infinite recursion.

Define descriptive scenario

Lastly to create a simulation, using the ScenarioBuilder you can define a scenario to simulate the historic performance of the example business and produce the descriptive insights.

Use ScenarioBuilder to build a scenario with the following steps:

  1. add a ScenarioBuilder for the Descriptive scenario
  2. add the resources and processes that are to be simulated in the scenario
  3. add the logic to end the simulation at the last sale date

In the code add the above changes:

// create a scenario to simulate historic business outcomes
const descriptive = new ScenarioBuilder("Descriptive")
    .resource(cash)
    .resource(inventory)
    .resource(liability)
    .resource(price)
    .resource(suppliers)
    .resource(reports)
    .process(sales)
    .process(receive_goods)
    .process(pay_supplier)
    .process(procurement)
    .process(reporter)
    // end the simulation at the last sale date
    .endSimulation(sales_dates.outputStreams().last)

Scenario Streams

Like any other builder, the ScenarioBuilder creates a task for Elara to run, which produces data stream outputs. In the case of the ScenarioBuilder, various streams are available through convenience methods, such as:

  • simulationJournalStream(): Return data stream containing a single journal detailing all the processes that occur during a simulation in chronological order.
  • simulationResultStreams(): Return an object of data streams containing the final simulation results, keyed by the name of the resource.

For example, to access the hourly reports generated in the simulation, you can use the simulationResultStreams method:

descriptive.simulationResultStreams().Reports

which is a tabular data stream with the following type, which could be used in a pipeline, function or visualisation, or even another resource, process!

DictType<StringType, StructType<{
    date: DateTimeType;
    cash: FloatType;
    liability: FloatType;
    inventory: IntegerType;
}>>

Define dashboard

Finally, to view the descriptive insights you can add a dashboard.

Define vega layouts

First, you can build visual layouts for the graphs of cash balance, inventory level, and liability over time using LayoutBuilder.

To build the cash graph, use the LayoutBuilder with the following steps:

  1. add a LayoutBuilder for the Cash graph
  2. add a vega graph with an appropriate title "Cash Balance vs Time"
    1. add a view to the graph
      1. add a fromStream to the view to define the chart from the Reports
      2. add a line to the view
        1. add an x value to the line using the date field
        2. add a y value to the line using the cash field

In the code add the above changes:

const cash_graph = new LayoutBuilder("Cash")
    .vega(
        "Cash Balance vs Time",
        builder => builder
            .view(builder => builder
                .fromStream(descriptive.simulationResultStreams().Reports)
                // create a line chart of the cash balance vs time, with a different color for each scenario
                .line({
                    x: builder => builder.value(fields => fields.date).title("Date"),
                    y: builder => builder.value(fields => fields.cash).title("Cash"),
                })
            )
    )

Repeat similar steps for inventory and liability.

Define dashboard layout

Finally, you can construct a dashboard layout that integrates all the vega graphs and add key performance indicators.

To define the dashboard layout, use the LayoutBuilder with the following steps:

  1. add a LayoutBuilder for the Dashboard layout
  2. add a tab to the layout
    1. add a layout to the tab for each graph
  3. add a header to the layout
    1. add an item to the header for each key performance indicator
      1. add a fromStream to the item to define the value from the simulationResultStreams
      2. add a value to the item to define how the value is displayed

In the code add the above changes:

const dashboard = new LayoutBuilder("Dashboard")
    .tab(builder => builder
        .layout(cash_graph)
        .layout(liability_graph)
        .layout(inventory_graph)
    )
    .header(
        builder => builder
            .item(
                "Price",
                builder => builder
                    .fromStream(descriptive.simulationResultStreams().Price)
                    .value((value) => PrintTruncatedCurrency(value),)
            )
            .item(
                "Profit",
                builder => builder
                    .fromStream(descriptive.simulationResultStreams().Cash)
                    .value((value) => PrintTruncatedCurrency(value),)
            )
            .item(
                "Inventory",
                builder => builder
                    .fromStream(descriptive.simulationResultStreams().Inventory)
                    .value((value) => value)
            )
            .item(
                "Liability",
                builder => builder
                    .fromStream(descriptive.simulationResultStreams().Liability)
                    .value((value) => PrintTruncatedCurrency(value))
            )
    )

Export solution

After defining all these components, compose everything into a Template to export as the entire solution. Use the export default statement to make the template available for deployment:

export default Template(/* list all components here */)

Congratulations, you have now created an end-to-end solution that incorporates, ingestion, pre-processing, simulation, and reporting.

Example Solution

The final solution for this tutorial is available below:

Next Steps

In the next lesson, you will learn about machine learning functions, and later you will learn about how to use them to build predictive insights.

On this page