Skip to main content

Define a pipeline operation

In this tutorial, you will define a pipeline operation in a pipeline.

This lesson will assume that you have an empty project and asset which you can to deploy to a workspace named 03_02_03_define_a_pipeline_operation with the following command:

edk template deploy -ycw 03_02_03_define_a_pipeline_operation

What are pipeline operations?

A pipeline operation performs a single transformation of data, and you define a pipeline operation within a pipeline.

You may define multiple pipeline operation in a sequence. Each defined operation in a pipeline performs a transformation on data and outputs a value (with the first operation performing the transformation on the source datastream defined via the from method). Each subsequent operation in the sequence performs a transformation on the preceding operation's output value. The pipeline's output datastream is populated with the result from the final operation of the pipeline.

There is a range of different kinds of operations that you can define in a pipeline. Common operation kinds that you will learn in upcoming tutorials include error(), innerJoin(), filter(), aggregate(), disaggregateArray() operations. Not all operation kinds can be performed on all values – the range of operations kinds possible on a value depends on the type of the value being transformed.

In this learning module, you will define a transform() operation, which is accessible for many types. You will learn more about other kinds of operations in later lessons.

Define a .transform() pipeline operation

A transform() operation in a pipeline generates a new value in an output datastream, where you define the calculation of the new value using expressions of the target value. You will learn more about expressions in the next lesson.

In an asset create a pipeline with a transform() operation:

import { SourceBuilder, PipelineBuilder, Template } from "@elaraai/core"

const my_source = new SourceBuilder("My Source")
    .value({ value: 2n })

const my_pipeline = new PipelineBuilder("My Pipeline")
    .from(my_source.outputStream())
    .transform(value => value)

export default Template(my_source, my_pipeline)
Note the use of function arguments

Many methods and Expreexpressionsssions in Elara require function arguments to define them. The exact function arguments available to be used as variables in a method/Expression depends on the lexical scope of the method/Expression you are defining. You can determine which function arguments are available in the lexical scope by hovering your mouse over the method or Expression in your project in your IDE.

For example, hovering your mouse over the transform() in a code editor will display the documentation explaining the use of the function arguments of the transform() method.

Transform the entire input Stream based on an EastFunction.

@param f — an EastFunction function that generates the output Expression

@returns — a new PipelineBuilder

@category — Pipeline

@example

 const username = Stream("Username", StringType);
 const password = Stream("Password", StringType);

 const pipeline = new PipelineBuilder("BasicAuth")
     .from(username)
     .input({ name: "password", stream: password })
     .transform((username, { password }) => StringJoin`${username}:${password}`)
     .toTemplate();

Don't worry if this pattern doesn't make any sense yet – you will learn the fundamentals behind this pattern and expressions in the next lesson.

In summary, you have defined a trivial transform() operation that does not change the solution from the previous tutorial – to do so, you define an Expression within the transform() operation.

Example solution

The code for this tutorial is available below:

Next steps

In this tutorial, you defined a trivial pipeline operation that passes an input datastream value through to an output datastream without mutating the value. In the next lesson, you will learn the fundamentals about expressions of datastreams and later apply them in your pipeline operation to transform data.

On this page