Pipeline - PipelineBuilder

Pipeline.PipelineBuilder

A PipelineBuilder to build a data

Pipeline.

Methods on the PipelineBuilder enable you to directly transform data from any

Stream by applying a transform operation. If the Stream is tabular, specialised operations can be applied to select, filter, join, aggregate, disaggregate and offset each entry in the Stream.

Quality of data can be observed using assert and warn by providing expression based conditions and messages to the whole

Stream.

A corresponding

Template can be created using .toTemplate().

Remarks

See

Transform Data for a related learning module.

Example

 const username = Stream("Username", StringType);
 const password = Stream("Password", StringType);

 const pipeline = new PipelineBuilder("BasicAuth")
     .from(username)
     .error({
         if: username => Equal(username, ""),
         message: "Username is empty"
     })
     .input({ name: "password", stream: password })
     .warn({
         if: (_, { password }) => Equal(password, ""),
         message: () => Const("Password is empty")
     })
     .transform((username, { password }) => StringJoin`${username}:${password}`)
     .transform(str => AsciiToBase64(str))
     .toTemplate();

Pipeline

constructor

• new PipelineBuilder(name, module?):

PipelineBuilder

Create a PipelineBuilder to build a data

Pipeline.

Methods on the PipelineBuilder enable you to directly transform data from any

Stream by applying a transform operation. If the Stream is tabular, specialised operations can be applied to select, filter, join, aggregate, disaggregate and offset each entry in the Stream.

Quality of data can be observed using assert and warn by providing expression based conditions and messages to the whole

Stream.

A corresponding

Template can be created using .toTemplate().

Parameters

Name	Type	Description
name	string	the name of the Pipeline to create
module?	ModulePath | ModuleBuilder	-

Returns

PipelineBuilder

Remarks

See

Transform Data for a related learning module.

Example

 const username = Stream("Username", StringType);
 const password = Stream("Password", StringType);

 const pipeline = new PipelineBuilder("BasicAuth")
     .from(username)
     .error({
         if: username => Equal(username, ""),
         message: "Username is empty"
     })
     .input({ name: "password", stream: password })
     .warn({
         if: (_, { password }) => Equal(password, ""),
         message: () => Const("Password is empty")
     })
     .transform((username, { password }) => StringJoin`${username}:${password}`)
     .transform(str => AsciiToBase64(str))
     .toTemplate();

---

from

▸ from(stream): T extends

DictType ? TabularPipelineBuilder : T extends BlobType ? BlobPipelineBuilder : GenericPipelineBuilder

Define the

Stream to construct the Pipeline from.

Type parameters

Name	Type
T	extends EastType

Parameters

Name	Type	Description
stream	Stream	the input Stream for the Pipeline

Returns

T extends

DictType ? TabularPipelineBuilder : T extends BlobType ? BlobPipelineBuilder : GenericPipelineBuilder

a new

PipelineBuilder

Example

 const username = Stream("Username", StringType);

 const pipeline = new PipelineBuilder("BasicAuth")
     .from(username)
     .toTemplate();