Skip to main content

Define a predictive scenario

In this tutorial you will learn how to create a scenario to simulate the future performance of the example business and produce predictive insights, you will:

  1. Add a pipeline to calculate an end date for the simulation
  2. Define ML models using the MLModelBuilder to infer demand and supplier choices, and train them on historic data.
  3. Add logic to key dates functions to predict future dates for sales and procurement, extend to use ml to predict future orders.
  4. Define a new resource to model a schedule of future orders, mapped from the procurement dates function output.
  5. Define processes for future sales and procurement, which can execute the existing processes.
  6. Create a predictive scenario to simulate future business outcomes, as a continuation of the past.
  7. Add a pipeline to concatenate the reports from both descriptive and predictive scenarios together.
  8. Build layouts to display orders and suppliers, and update charts to use the concatenated reports
  9. Construct a dashboard that combines the new layout into a panel, and uses predictive values for the header

This lesson will assume that you have an empty project and asset which you can to deploy to a workspace named 04_03_03_define_a_predictive_scenario with the following command:

edk template deploy -ycw 04_03_03_define_a_predictive_scenario

Define new constants

Along with the recommended retail price, there are two more constants that you need to define. The opening and closing hours of the business. These will be used to calculate future sales dates.

const opening_hour = 9n;
const closing_hour = 17n;

Define ml functions

To predict things unknown values, you can leverage MLModelBuilder to define machine learning models. You can define two models for the example business, the first model will predict demand based on price, day of the week, and hour of the day. The second model will infer supplier choices based on the day of the week.

Define demand model

You can use the MLModelBuilder to define a machine learning model to predict the sales demand with the following steps:

  1. create a new MLModelBuilder with a name
  2. add features to the model using feature method, these are the inputs to the model
    1. The first feature is the price, which is a FloatType
    2. The second feature is the day of the week, which is an IntegerType
    3. The third feature is the hour of the day, which is an IntegerType
  3. define the output type of the model using the output method
  4. train the model using the trainFromPipeline method, which takes a pipeline builder as an argument
    1. define the name from the data which represents the output of the model, in this case "qty"
    2. add a selection operation to the pipeline, to calculate the features and output

In the code add the above changes:

// define an ml model to infer demand, from price and time in week
const historic_demand = new MLModelBuilder("Demand")
    // add features to the model
    .feature("price", FloatType)
    .feature("dayOfWeek", IntegerType)
    .feature("hourOfDay", IntegerType)
    // the output is qty which is FloatType
    .output(FloatType)
    // define the historic data as the training set
    .trainFromPipeline({
        output_name: "qty",
        pipeline: builder => builder
            .from(sales_data.outputStream())
            // select the features and output from fields
            .select({
                selections: {
                    price: fields => fields.price,
                    // calculate the day of week and hour of day from the date
                    dayOfWeek: fields => DayOfWeek(fields.date),
                    hourOfDay: fields => Hour(fields.date),
                    // convert the qty to a float
                    qty: fields => Convert(fields.qty, FloatType),
                }
            })
    })
Model Training

By using trainFromPipeline, Elara will create a task that will automatically train the model when the data changes (in this case sales_data). This will ensure that the model is always up to date with the latest data.

Elara uses the same optimizer that optimizes the business, to automatically build the ml models, with consideration of the output data type, the data (split into training and testing set automatically) as well as model execution time.

An important concept about ML in Elara, is that the goal is not to infer the maximum likelihood of the output, but rather to infer the distribution of the output. This is because the output of the model is used to simulate the business, and the business is stochastic. Therefore, the model should be stochastic too.

Define supplier choice model

You can also use the MLModelBuilder to define a machine learning model to predict the supplier choice in an order with the following steps:

  1. create a new MLModelBuilder with a name
  2. add features to the model using feature method, these are the inputs to the model
    1. The first feature is the day of the week, which is an IntegerType
  3. define the StringType output of the model using the output method
  4. train the model using the trainFromPipeline method, which takes a pipeline builder as an argument
    1. define the name from the data which represents the output of the model, in this case "supplierName"
    2. add a selection operation to the pipeline, to calculate the features and output

In the code add the above changes:

// define an ml model to infer the typical supplier choice on a day
const historic_supplier_choice = new MLModelBuilder("Supplier Choices")
    // add a feature to the model
    .feature("dayOfWeek", IntegerType)
    // the output is supplier name which is StringType
    .output(StringType)
    // define the historic data as the training set
    .trainFromPipeline({
        output_name: "supplierName",
        pipeline: builder => builder
            .from(procurement_data.outputStream())
            // select the feature and output from fields
            .select({
                selections: {
                    // calculate the day of week from the date
                    dayOfWeek: fields => DayOfWeek(fields.date),
                    supplierName: (fields) => fields.supplierName,
                }
            })
    })

Define future key dates calculation

Previously the simulation only recreated a historic period of time from the historic data. To create predictive insights you need to simulate into the future. To do this you need to define a function to calculate the next sales date, and a function to calculate the next procurement dates.

Extend sales dates function

You can extend the existing definition of the Sales Dates function to calculate the next sales date. To do this you need to:

  1. add a new variable to the function, which is the next sales date accounting for the opening and closing hours
  2. add a new output to the function, which is the next sales date

In the code add the above changes:

// Generate the keys dates related to sales
const sales_dates = new FunctionBuilder("Sales Dates")
    .input("sales", sales_data.outputStream())
    .body(builder => builder
        // when the last sale hour occurred (this is the current date time)
        .let("last", vars => Reduce(
            vars.sales,
            (prev, sale) => Max(prev, GetField(sale, 'date')),
            DefaultValue(DateTimeType))
        )
        // when the first sale hour occurred (historically)
        .let("first", vars => Reduce(
            vars.sales,
            (prev, sale) => Min(prev, GetField(sale, 'date')),
            vars.last
        ))
        // when the next sale hour will occur (historically)
        .let("next", vars => IfElse(
            Less(Hour(vars.last), closing_hour),
            AddDuration(vars.last, 1, 'hour'),
            AddDuration(Ceiling(vars.last, 'day'), opening_hour, 'hour'),
        ))
        // return all the dates
        .return({
            last: vars => vars.last,
            first: vars => vars.first,
            next: vars => vars.next
        })
    )

Define end date calculation

To simulate into the future, you need to determine a future end date. You use a PipelineBuilder to calculate the end date, with the following steps:

  1. create a new PipelineBuilder with a name
  2. add a from operation to the pipeline, to get the next sales date from the Sales Dates function
  3. add a transform operation to the pipeline, to calculate the end date as the next sales date plus one week, rounded to the end of the week

In the code add the above changes:

// calculate the future date to simulate until
const end_date = new PipelineBuilder("End Date")
    .from(sales_dates.outputStreams().next)
    .transform(date => Ceiling(AddDuration(date, 1, 'week'), 'day'))

Define procurement dates function

To predict the procurement, you can define the relevant date ranges, as well as pre-calculated schedule of orders. To do this you need to:

  1. create a new FunctionBuilder with a name
  2. add inputs to the function, which are the end date and the historic procurement data
  3. add the historic_supplier_choice model to infer the supplier name
  4. add a body statement to the function, to calculate the next procurement date and schedule of orders
    1. add a let statement, to get the last order date and time
    2. add a let statement, to get the date of the next order date and time
    3. add a let statement, to set the scheduled date to the next order date and time
    4. add a let statement, to create an empty schedule of orders
    5. add a while statement, to continue until the scheduled date is after the end date
      1. add a let statement, to infer the supplier name using the historic_supplier_choice model
        1. use the DayOfWeek expression to get the day of the week from the scheduled date for the model input
      2. add an insert statement, to insert the date into the schedule
      3. add an assign statement, to increment the date to the next day
  5. add a return statement, to return the last order date and time, the next order date and time, and the schedule of orders

In the code add the above changes:

// generate the procurement dates, and a schedule of orders
const procurement_dates = new FunctionBuilder("Procurement Dates")
    .input("end", end_date.outputStream())
    .input("procurement", procurement_data.outputStream())
    .ml(historic_supplier_choice)
    .body(block => block
        // get the last order date and time
        .let("last", vars => Reduce(
            vars.procurement,
            (prev, order) => Max(prev, GetField(order, 'date')),
            DefaultValue(DateTimeType))
        )
        // get the date of the next order date and time
        .let("next", vars => AddDuration(vars.last, 1, 'day'))
        .let("scheduled", vars => vars.next)
        .let("schedule", () => NewDict(StringType, StructType({ date: DateTimeType, supplierName: StringType })))
        // start at the next order date and time
        .while(
            // continue until the end date
            vars => LessEqual(vars.scheduled, vars.end),
            (block) => block
                // infer the supplier name using the ml function
                .let("supplierName", (vars, mls) => mls['Supplier Choices'](Struct({
                    dayOfWeek: DayOfWeek(vars.scheduled)
                })))
                // insert the date into the schedule
                .insert(
                    vars => vars.schedule,
                    // print the date as a string with format YYYY-MM-DD
                    vars => Print(vars.scheduled, "YYYY-MM-DD"),
                    // add an order into the schedule, with the inferred supplier name
                    vars => Struct({
                        date: vars.scheduled,
                        supplierName: vars.supplierName,
                    })
                )
                // increment the date to the next day
                .assign("scheduled", vars => AddDuration(vars.scheduled, 1, 'day'))
        )
        // return all the dates and schedule
        .return({
            last: vars => vars.last,
            next: vars => vars.next,
            schedule: vars => vars.schedule
        })
    )

Define future order resource

The schedule of orders is a resource that can be used to simulate future procurement, and create with the following steps:

  1. create a new ResourceBuilder with a name
  2. add a mapFromStream operation to the resource, to map the schedule of orders from the Procurement Dates function output

In the code add the above changes:

// create a resource containing procurement schedule
const orders = new ResourceBuilder("Orders")
    .mapFromStream(procurement_dates.outputStreams().schedule)

Define predicted processes

Now that you have the future dates, and order schedule, you can define processes to simulate future sales and procurement.

Define predicted sales process

You can use the ProcessBuilder to create a process that will simulate future sales, with the following steps:

  1. create a new ProcessBuilder with a name
  2. add resources to the process, which are the inventory and price
  3. add the Sales process to the Predicted Sales process
  4. add the historic_demand model to infer the demand
  5. add a let statement, to calculate the demand with the Demand function
    1. use the DayOfWeek expression to get the day of the week from the date for the model input
    2. use the Hour expression to get the hour of the day from the date for the model input
  6. add an execute statement, to execute the Sales process
    1. use the Struct expression to create a new struct with the date, demand, and price
  7. add an execute statement, to execute the Predicted Sales process an hour later, or at the beginning of the next day
  8. add a mapFromPipeline statement to the process, to start simulating from the opening hour of the first sales period in the future

In the code add the above changes:

// create sales in the future, inferring demand based on price and time in day and week
const predicted_sales = new ProcessBuilder("Predicted Sales")
    .resource(inventory)
    .resource(price)
    .process(sales)
    .ml(historic_demand)
    // calculate the demand with the ml function
    .let("demand", (props, resources, mls) => Ceiling(mls.Demand(Struct({
        price: resources.Price,
        dayOfWeek: DayOfWeek(props.date),
        hourOfDay: Hour(props.date)
    })), "integer"))
    // execute the sales process
    .execute("Sales", (props, resources) => Struct({
        date: props.date,
        qty: props.demand,
        price: resources.Price
    }))
    // create another sale in an hour, taking into account the opening and closing hours
    .execute("Predicted Sales", (props) => Struct({
        // the next sale date will be in an hour, or the beginning of the next day
        date: IfElse(
            Less(Hour(props.date), closing_hour),
            AddDuration(props.date, 1, 'hour'),
            AddDuration(Ceiling(props.date, 'day'), opening_hour, 'hour'),
        )
    }))
    // start simulating from the opening hour of the first sales period in the future
    .mapFromPipeline(builder => builder
        .from(sales_dates.outputStreams().next)
        .transform(date => Struct({ date }))
    )

Define predicted procurement process

You can define the predicted procurement process, with the following steps:

  1. create a new ProcessBuilder with a name
  2. add resources to the process, which are the orders
  3. add the Procurement process to the Predicted Procurement process
  4. add a forDict statement, to iterate over the predicted orders
    1. use the GetField expression to get the date from the order
    2. use the GetField expression to get the supplier name from the order
    3. add an execute statement, to execute the Procurement process
      1. use the Struct expression to create a new struct with the date and supplier name
  5. add a mapFromPipeline statement to the process, to start simulating from the last procurement date

In the code add the above changes:

// create procurement in the future, based on the procurement schedule
const predicted_procurement = new ProcessBuilder("Predicted Procurement")
    .resource(orders)
    .process(procurement)
    .forDict(
        (_props, r) => r.Orders,
        (block, order) => block
            .execute("Procurement", () => Struct({
                date: GetField(order, 'date'),
                supplierName: GetField(order, 'supplierName'),
            }))
    )
    // start simulating from the last procurement date, since this will be prior to the next procurement date in the schedule
    .mapFromPipeline(builder => builder
        .from(procurement_dates.outputStreams().last)
        .transform(date => Struct({ date }))
    )
Order prediction

Note that for demonstration purposes, the orders were pre-created in the Procurement Dates function. In a real world scenario, may infer the supplierName in place in the Predicted Procurement process to create the orders.

Define predictive scenario

Finally, the predicted processes and order resource can be combined into a scenario to simulate future business outcomes. Since we already have a scenario simulating the past, we can continue the simulation from the end of the descriptive scenario tpo simulate the future. To do this you need to:

  1. create a new ScenarioBuilder with a name
  2. define the Predictive scenario as the continuation of the Descriptive scenario
  3. add resources to the scenario, which are the predicted orders
  4. add processes to the scenario, which are the predicted sales and procurement processes
  5. add an alterResourceFromValue statement to the scenario, to clear the reports
  6. add an endSimulation statement to the scenario, to end the simulation at the end date

In the code add the above changes:

// create a scenario to simulate future events
const predictive = new ScenarioBuilder("Predictive")
    .continueScenario(descriptive)
    .resource(orders)
    .process(predicted_sales)
    .process(predicted_procurement)
    .alterResourceFromValue("Reports", new Map())
    // end simulation at the end date
    .endSimulation(end_date.outputStream())
Scenario continuation

With a continued scenario, the goal is to safely continue a simulation, this means all processes and resources are "copied", including their values at the end of the continued simulation.

Aside form resources and processes, another stream can demonstrate the benefit of continuation and can be accessed with a convenience method on the scenario builder, simulationQueueStream.

  • simulationQueueStream(): Return a stream containing an array of all the processes pending at the end of simulation in priority queue order.

For example in the above example, due to the fact that we expect historic procurement to occur before the end of the Descriptive scenario, it's possible that it may trigger another process such as Receive Goods, or Pay Supplier AFTER the simulation end date. It would be bad practice to ignore those future processes since they effect important resources; continuing the Descriptive scenario ensures that any processes triggered that didn't occur appear in the simulationQueueStream, and therefore will be executed in the Predictive scenario.

Resource alteration

With a scenario it's possible to alter the resource values that the simulation starts with, this is useful for example to clear the reports, or to set the initial cash balance. There are multiple methods to alter resources:

  • alterResourceFromValue(): Replace the initial value of a resource in this scenario with a provided value.
  • alterResourceFromStream(): Replace the initial value of a resource in this scenario with a value from a data stream.
  • alterResourceFromPipeline(): Replace the initial value of a resource in this scenario with the output of a new pipeline.

Define report concatenation

To show both the descriptive and predictive scenarios on the same dashboard, you can concatenate the reports from both scenarios together. To do this you need to:

  1. create a new PipelineBuilder with a name
  2. add a from operation to the pipeline, to get the reports from the Descriptive scenario
  3. add a filter operation to the pipeline, to filter the reports to only include the last week of the simulation
    1. add an input to the pipeline, which is the next sales date from the Sales Dates function
    2. use the GreaterEqual expression to filter the reports to only include the last week of the simulation
  4. add an input to the pipeline, which is the reports from the Predictive scenario
  5. add a concatenate operation to the pipeline, to concatenate the reports from both scenarios together
    1. add a discriminator_name to the concatenate operation, as a scenario
    2. add a discriminator_value to the concatenate operation, which is the scenario name
    3. add an inputs to the concatenate operation, which is the reports from the Predictive scenario

In the code add the above changes:

// combine the reports from all multiple scenarios
const concatenated_reports = new PipelineBuilder("Concatenated Reports")
    .from(descriptive.simulationResultStreams().Reports)
    // filter the historic data to include one week in the past
    .input({ name: "next", stream: sales_dates.outputStreams().next })
    .filter((fields, _key, inputs) => GreaterEqual(fields.date, SubtractDuration(inputs.next, 1, 'week')))
    // combine the reports from the future scenarios
    .input({ name: "BAU", stream: predictive.simulationResultStreams().Reports })
    .concatenate({
        discriminator_name: "scenario",
        discriminator_value: "Historic",
        inputs: [
            { input: inputs => inputs.BAU, discriminator_value: "BAU" },
        ]
    })

Update dashboard

Finally, you can update the dashboard to display the orders and suppliers, and update the charts to use the concatenated reports.

Define order and supplier layouts

First add table visuals for displaying the new predicted orders, with the following steps:

  1. create a new LayoutBuilder with a name
  2. add a table visual to the layout, to display the orders
    1. add a fromStream operation to the table, to get the orders from the Orders resource
    2. add a date column to the table, which is the date from the order
    3. add a string column to the table, which is the supplier name from the order

In the code add the above changes:

// create an editable table of orders
const orders_graph = new LayoutBuilder("Orders")
    .table(
        "Orders",
        builder => builder
            .fromStream(predictive.simulationResultStreams().Orders)
            .date("Date", fields => fields.date)
            .string("Supplier Name", fields => fields.supplierName)
    )

Repeat similar steps for the Suppliers from predictive scenario.

Update vega layouts

You can change the charts to use the new concatenated reports, with the following steps:

  1. add a color scale to apply, based on the scenario
  2. change the fromStream in the Cash LayoutBuilder to use the Concatenated Reports pipeline
  3. add the color encoding to use the scenario field from the reports, and the new color scale
// make some colors for each scenario in the charts
const colors: [value: string, color: string][] = [['Optimized', '#2B4B55'], ['BAU', '#6da7de'], ['Historic', '#b5bac0']]

// the graphs for the report values
const cash_graph = new LayoutBuilder("Cash")
    .vega(
        "Cash Balance vs Time",
        builder => builder
            .view(builder => builder
                .fromStream(concatenated_reports.outputStream())
                // create a line chart of the cash balance vs time, with a different color for each scenario
                .line({
                    x: builder => builder.value(fields => fields.date).title("Date"),
                    y: builder => builder.value(fields => fields.cash).title("Cash"),
                    color: builder => builder.value(fields => fields.scenario).scale(colors)
                })
            )
    )

Repeat similar steps for the Inventory and Liability charts.

Congratulations, you have now created an end-to-end solution that incorporates, ingestion, pre-processing, simulation, ml, prediction and reporting.

Example Solution

The final solution for this tutorial is available below:

Next Steps

In the next lesson, you will learn how to optimize and create prescriptive insights.

On this page