16th December 2024

Data pipelines - full release

Full release of data pipelines, coming out of the BETA phase. https://knowledge.curiositysoftware.ie/docs/pipelines

Query Composer

1. In-product database interactive window. This allows users to connect to their exposed database and run SQL queries.
   a. Location within the side menu

2. Database connection selector

3. Table view of chosen database

4. Query editor

5. Query results
   

API Composer

Allow users to compose API calls, either from scratch or by uploading an existing JSON that describes a call. These API calls can then be saved, or exposed to a Data Pipeline.

1. Location within the side menu

2. Method

3. URL

4. API call options

5. Body for Post  

Skip number of records in find and control limit

Two new options in a Find job, these are set when you create the find submission form.
 SQL Limit - restricts the number of rows that the job can return, set to 500 by default.

Skip Rows - Setting this will skip the first X records in the find job, 0 by default.

Data Generation quickstart wizard

When creating a new data generation activity, there is now a quickstart wizard that allows users to add in required assets to the activity.
The Wizard will ask for Activity Details, a Definition, and a Rule Set. If users already have a ruleset, then you have the choice to prepare the requisite infrastructure to run the generate, and create the reusable submission form.

Job progress and result popup

When triggering a new job, a small job dialogue will popup in the bottom corner of the screen. This replaces the existing behaviour when a job would open in a new tab.