Pipeline - GenericPipelineBuilder

Pipeline.GenericPipelineBuilder

A GenericPipelineBuilder to build a data

Pipeline.

Methods on the GenericPipelineBuilder enable you to directly transform data from any

Stream by applying a transform operation.

Quality of data can be observed using assert and warn by providing expression based conditions and messages to the whole

Stream.

A corresponding

Template can be created using .toTemplate().

Remarks

See

Transform Data for a related learning module.

Example

 // get a predefined pipeline table
 const username = Stream("Username", StringType);
 const password = Stream("Password", StringType);

 const pipeline = new PipelineBuilder("BasicAuth")
     .from(username)
     .error({
         if: username => Equal(username, ""),
         message: "Username is empty"
     })
     .input({ name: "password", stream: password })
     .warn({
         if: (_, { password }) => Equal(password, ""),
         message: () => Const("Password is empty")
     })
     .transform((username, { password }) => StringJoin`${username}:${password}`)
     .transform(str => AsciiToBase64(str))
     .toTemplate();

Type parameters

Name	Type
Output	extends EastType
Inputs	extends Record

Hierarchy

• Builder

  ↳ GenericPipelineBuilder

Pipeline

error

▸ error(config):

GenericPipelineBuilder

Add an assertion on the pipeline inputs and output to identify errors. When the if predicate returns true the pipeline will be terminated with an error message and output data will not be produced.

Parameters

Name	Type	Description
config	Object	the error message and predicate
config.if	(value: Variable, inputs: Inputs) => EastFunction	If true an error will be created
config.message	string | (value: Variable, inputs: Inputs) => EastFunction	The message in the case that an error is created

Returns

GenericPipelineBuilder

Example

 const username = Stream("Username", StringType);
 const password = Stream("Password", StringType);

 const pipeline = new PipelineBuilder("BasicAuth")
     .from(username)
     .error({
         if: username => Equal(username, ""),
         message: "Username is empty"
     })
     .input({ name: "password", stream: password })
     .transform((username, { password }) => StringJoin`${username}:${password}`)
     .error({
         if: str => Equal(str, ":"),
         message: () => Const("Unexpected string")
     })
     .toTemplate();

---

input

▸ input(config):

GenericPipelineBuilder<Output, Inputs & { [K in string]: Variable }>

Add an additional named input

Stream to the Pipeline.

Type parameters

Name	Type
Name	extends string
T	extends EastType

Parameters

Name	Type	Description
config	Object	the input stream and the resulting variable name
config.name	Name extends "input" | keyof Inputs ? never : Name	-
config.stream	Stream	-

Returns

GenericPipelineBuilder<Output, Inputs & { [K in string]: Variable }>

a new

PipelineBuilder

Example

 const username = Stream("Username", StringType);
 const password = Stream("Password", StringType);

 const pipeline = new PipelineBuilder("BasicAuth")
     .from(username)
     .input({ name: "password", stream: password })
     .toTemplate();

---

let

▸ let(name):

GenericPipelineBuilder<Output, Inputs & { [K in string]: Variable }>

Give a name to the current

Pipeline output to be used as an input later in the Pipeline (i.e. after the next operation)

Type parameters

Name	Type
Name	extends string

Parameters

Name	Type
name	Name extends "input" | keyof Inputs ? never : Name

Returns

GenericPipelineBuilder<Output, Inputs & { [K in string]: Variable }>

a new

PipelineBuilder

Example

 const username = Stream("Username", StringType);
 const password = Stream("Password", StringType);

 const pipeline = new PipelineBuilder("BasicAuth")
     .from(username)
     .let("username")
     .input({ name: "password", stream: password })
     .transform((username, { password }) => StringJoin`${username}:${password}`)
     .transform((str, inputs) => Struct({
         Username: inputs.username,
         Hash: str
     }))
     .toTemplate();

---

log

▸ log(config):

GenericPipelineBuilder

Produce a log message depending on the pipeline inputs and ouputs. When the if the predicate returns true the pipeline will produce a log message and proceed.

Parameters

Name	Type	Description
config	Object	the log message and optional predicate
config.if?	(value: Variable, inputs: Inputs) => EastFunction	If true a log message will be produced (optional)
config.message	string | (value: Variable, inputs: Inputs) => EastFunction	The log message

Returns

GenericPipelineBuilder

Example

 const username = Stream("Username", StringType);
 const password = Stream("Password", StringType);

 const pipeline = new PipelineBuilder("BasicAuth")
     .from(username)
     .log({
         if: username => Equal(username, ""),
         message: "Username is empty"
     })
     .input({ name: "password", stream: password })
     .transform((username, { password }) => StringJoin`${username}:${password}`)
     .log({
         if: str => Equal(str, ":"),
         message: () => Const("Unexpected string")
     })
     .toTemplate();

---

outputStream

▸ outputStream():

Stream

Return the

Stream containing the output of the pipeline.

Returns

Stream

Example

// Create a datastream where a password can be written by end-user at runtime.
const hourly = new SourceBuilder("DatabasePassword")
    .writeable(StringType)
    .outputStream()

---

toJson

▸ toJson(config):

BlobPipelineBuilder

Unparse a

Stream into a BlobType Stream containing JSON data.

Type parameters

Name	Type
S	extends (value: Variable, inputs: Inputs) => EastFunction

Parameters

Name	Type	Description
config	Object	the configuration of the JSON parsing
config.value	S	the function to parse the value

Returns

BlobPipelineBuilder

a new

PipelineBuilder

Example

 const username = Stream("Username", StringType);

 const pipeline = new PipelineBuilder("BasicAuth")
     .from(username)
     .toJson({
         value: (value) => value
     })
     .toTemplate();

---

toTemplate

▸ toTemplate():

Template

Convert the built pipeline into an

Template, for usage in an EDK project.

Returns

Template

a

Template

Example

 const username = Stream("Username", StringType);

 const template = new PipelineBuilder("BasicAuth")
     .from(username)
     .toTemplate();

Overrides

Builder.toTemplate

---

transform

▸ transform(f): ReturnType["type"] extends

DictType ? TabularPipelineBuilder : GenericPipelineBuilder<ReturnType["type"], Inputs>

Transform the entire input

Stream based on an EastFunction.

Type parameters

Name	Type
F	extends (value: Variable, inputs: Inputs) => EastFunction

Parameters

Name	Type	Description
f	F	an EastFunction function that generates the output Expression

Returns

ReturnType["type"] extends

DictType ? TabularPipelineBuilder : GenericPipelineBuilder<ReturnType["type"], Inputs>

a new

PipelineBuilder

Example

 const username = Stream("Username", StringType);
 const password = Stream("Password", StringType);

 const pipeline = new PipelineBuilder("BasicAuth")
     .from(username)
     .input({ name: "password", stream: password })
     .transform((username, { password }) => StringJoin`${username}:${password}`)
     .toTemplate();

---

warn

▸ warn(config):

GenericPipelineBuilder

Add a warning on the pipeline inputs and ouputs to identify problems. When the if predicate returns true the pipeline will register a warning with a message, but will proceed to proceed to produce output data.

Parameters

Name	Type	Description
config	Object	the warning message and predicate
config.if	(value: Variable, inputs: Inputs) => EastFunction	If true a warning will be produced
config.message	string | (value: Variable, inputs: Inputs) => EastFunction	The message in the case that a warning is produced

Returns

GenericPipelineBuilder

Example

 const username = Stream("Username", StringType);
 const password = Stream("Password", StringType);

 const pipeline = new PipelineBuilder("BasicAuth")
     .from(username)
     .warn({
         if: username => Equal(username, ""),
         message: "Username is empty"
     })
     .input({ name: "password", stream: password })
     .transform((username, { password }) => StringJoin`${username}:${password}`)
     .warn({
         if: str => Equal(str, ":"),
         message: () => Const("Unexpected string")
     })
     .toTemplate();