What is a pipeline?
A pipeline is a visual workflow that allows you to automate and manage data processes by connecting activities and utility elements together on a canvas. Pipelines can be used to process, transform, move, generate, mask, and subset data, enabling complex tasks to be executed as a single automated workflow.
By defining the sequence of activities and decision points, pipelines help reduce manual effort, improve consistency, and provide a clear view of how data flows through a process. Pipelines can be used for both development and production use cases and support a wide range of data management and automation requirements.
Navigation and File Management
Use this toolbar to navigate between pages, save your work, and access additional pipeline options and settings.
Button icon | Button Description |
|---|---|
| Shortcut to dashboard Returns you to the Activity Dashboard where you can view, manage, and access activities and pipelines |
| Saves pipeline Saves any changes made to the current pipeline. It is recommended to save your work regularly while building or editing a pipeline. |
| Data activities expanded Displays the list of available data activities that can be added to the pipeline canvas. |
| More options: Provides access to additional pipeline functions, including:
|
Pipeline Execution and Editing
Use this toolbar to run and debug the pipeline, manage variables, and perform common editing actions such as undo, redo, copy, cut, and paste.
Button Icon | Button Description |
|---|---|
| Run Pipeline The button to run/execute the pipeline Executes the pipeline from the configured start node and processes all connected activities according to the defined workflow. |
| Pipeline variables Manage pipeline variables that can be used to store and pass values between activities during execution. You can add, edit and remove pipeline variables here. |
| Undo Undo Reverses the most recent action performed on the canvas. |
| Redo Restores the last action that was undone. |
| Copy element Creates a copy of the selected pipeline element for reuse elsewhere in the pipeline. |
| Cut element Removes the selected element from the canvas and places it on the clipboard ready to be pasted elsewhere. |
| Debug Pipeline Runs the pipeline in debug mode, allowing you to review execution details, identify issues, and verify pipeline behaviour before running it in production. |
| More Options
|
| Properties Panel Displays the configuration settings and properties of the selected node or connection. |
| Parameters Panel Allows you to create, edit, and manage parameters associated with a selected node or connection. |
| View Run Steps and Logs Displays pipeline execution steps, status information, and logs that can be used to troubleshoot issues and monitor processing. |
Canvas Elements and Configuration
Contains the available elements that can be added to a pipeline. You can drag and drop elements directly onto the canvas to build your workflow.
Creating Connections Select an element on the canvas to display directional arrows. Click and drag from an arrow to another element to create a connection. Connections determine how the pipeline flows from one step to the next during execution.
Button Icon | Button Description |
|---|---|
| Start node Defines the point at which pipeline execution begins. Every pipeline must contain a start node. |
| End node Defines the point at which pipeline execution completes. |
| Assign Element (Quick Add) Adds an Assign element used to create or update variable values within the pipeline. |
| Decision Element (Quick Add) Adds a Decision element used to evaluate conditions and direct the pipeline down different execution paths. |
| Invoke Process Element (Quick Add) Adds an Invoke Process element used to execute another process, activity, or pipeline as part of the workflow. |
| Log Message Element (Quick Add) Adds a logging element that writes custom messages to the execution log to assist with monitoring and debugging. |
| Text Element (Quick Add) Adds a text annotation to the canvas that can be used to document pipeline logic or provide additional information. |
|
Adds a visual note to the canvas for documenting design decisions, reminders, or explanations for other users. |
