Supported Database Types

Data Subsetting from Test Data Automation supports the following database types:

  • MySQL

  • Postgres

  • SQL Server

  • Oracle

  • DB2

Prerequisite Knowledge and Technologies

To use Data Subsetting from Test Data Automation, you will need an intermediate understand of SQL. You will also need a good level of proficiency in Microsoft Excel, and an understanding of the database that you are subsetting.
There are four pieces of prerequisite software for data subsetting:

  1. You must be running a 64-bit Windows machine. Windows 10 is strongly recommended and is required for the full range of functionality.

  2. The latest 64-bit version of Microsoft Excel.

  3. The latest version of VIP. Information on prerequisites and installation can be found here, and licensing information here.

  4. The latest version of the VIP Execution Server. Prerequisites and installation can be found here.

Once you have installed the prerequisite software on a 64-bit Windows machine, you will require licensed access to the relevant TDM utilities. Contact your account manager at Curiosity if you require a licence, or email Info@Curiosity.Software.

How to Copy the Requisite Files into a Subset Folder

All the files used to set up a Subset job are shipped during the installation of the VIP Server Controller.

You will not need to edit most of these files. However, it is useful to understand which files are used during the automated Subsetting Actions. This is valuable when debugging failed Subsetting Actions as a parameter might be pointing to the wrong location. The location of all files is therefore summarised below.

Understand and locate the files used during Subsetting

If you followed the standard installation, these folders will be stored on your C: Drive.

You will find seven folders. Each folder's name is prefaced with "VIP":

C:\VIPTDM contains the files used to configure Test Data Automation utilities, including data subsetting.

The files used during subsetting are separated into three folders. One contains the files you will edit and work with. The other contains files that will remain static.

It is strongly recommended that you leave these files in the default location if possible, or at least that they remain in one location. This will let you use many of the same parameters throughout the automated Subset Actions. If you move these files between Actions, you will need to update several additional parameters.

  1. C:\VIPTDM\SubsetCloneAndMask – Contains files that you will not edit

This folder contains the generic automated workflows used to perform the actions. Do not edit files in this folder. If possible, the location of this folder will should also remain the same.
The files contained in this folder are:

  1. The automated workflows used to perform the Subset Actions:

    1. SQLSubset.enc.vip

    2. SQLActions.enc.vip

  2. The .DLL Files associated with the automated Workflows.

  3. The .XML files associated with the automation.

  4. TXT Files associated with the automation. 

Note: The VIP flows are encrypted and you need a valid subsetting licence to run them. If you do not have licensed copies of the above files, please email your Curiosity account manager or email info@Curiosity.Software.

  1. C:\VIPTDM\SubsetCloneAndMaskRun

 – Contains files that you will not edit

This folder contains additional static files used when performing Subsetting, Cloning and Masking. The file relevant to Subsetting is:

  1. A FlowCache.txt – this is used to tell the VIP Server Controller the location of VIP workflows to put "in memory". This improves the performance of the automation.

The folder also contains subfolders for running a "Mask", "Subset" or "Clone". These folders contain the files you will need to edit.

  1. The "C:\VIPTDM\SubsetCloneAndMaskRun\Subset" 

– This is your "Run Directory" It contains files that you should Copy and edit

This folder contains the files you will edit when running a Subset (see below).

Create a new Run Directory

C:\VIPTDM\SubsetCloneAndMaskRun\Subset" contains the files you will edit when running a Subset (see below).

This folder is grouped by different data sources. Select the type of data which you wish to Subset:

This folder contains the configuration files that you should edit. These files are:

  1. A blank Basic Control Spreadsheet that points the VIP workflow to the database that will be modelled and subsetted. This is called "VIPSubset" followed by the name of the data type.

  2. cfg – A "Configuration File" that will load a set of default parameters into The VIP Server Controller. This allows you to quickly edit a set of pre-set parameters and also provides the default location of the static, unedited files used during Subsetting.

  3. The standard scripts used to execute the Subset Actions. These scripts are used to parameterize and execution the Subset Actions if executing Actions from the Command Line instead of the VIP Server Controller. The scripts are:

    1. Subset1_MetaData.cmd

    2. Subset2_Prep.cmd

    3. Subset3_Build.cmd

    4. Subset4_Subset.cmd

Note: the file name of each script reflects the action it performs. The numeration reflects the typical order in which you would run the actions when iterating Subsets.
It is recommended that you create a new "Run Directory" folder for each individual Subset. This enables you to easily return to edit individual subsetting jobs. It also avoids unnecessary rework, helping to avoid previously configured Subsets being edited or lost. You might additionally incorporate a file structure where each Subset folder is stored under individual folders for individual Databases or Schemas.

Start by copying all files from the relevant data type folder to a new folder. Below, you can see an example copied from C:\VIPTDM\SubsetCloneAndMaskRun\Subset\SQLServer:

Overview of Methods used to Run Subset Actions

Data Subsetting

TEST DATA AUTOMATION DATA SUBSETTING REQUIREMENTS