What is a pipeline?
Pipelines transform data through a series of calculations, or operations. They are one way of to automate processing of data with a task on Elara.
A pipeline in a solution:
- takes one or more datastreams as input
- undertakes a series of transformation operations that you define,
- outputs the resulting value in a separate datastream, and
- performs the operations each time that the input data changes, so values are up-to-date with changing input values.
A pipeline combines and/or transforms data from one form to another, while preserving the state of the input datastream(s). The output does not need to share the same data type as the input.
Pipelines are a declarative interface for transforming data. The most common declarative language for data transformation is SQL. In SQL you declare what result you desire, and it is up to the database how to best perform the query. Similarly, you do need to tell Elara how to perform pipeline operations. They are a higher-level interface which is easier to use than writing an imperative program would be.
What is a task?
It may be helpful to think of how pipelines operate on the Elara platform with an analogy to Microsoft Excel. Some cells in Excel contain data entered by the user, much like the writeable data sources studied in the previous module. Other cells in Excel may contain formulae, which perform calculations based on the values in other cells. These calculations are kept up-to-date automatically by Excel when the input data changes.
Pipelines are just one kind of "task" on the Elara platform. Tasks perform calculations based on the values in input datastreams, and write the results to output datastreams. Like Excel, Elara will automatically recalculate values by rerunning tasks whenever input data changes.
Other examples of tasks on Elara are "functions", "simulations" and "machine learning training". You will learn to use other types of tasks later.
Next steps
In the next tutorial, you will define a pipeline and launch a pipeline on Elara.