Machine Learning Further Information

Analyze

Analyze reads your training data before you train. It checks the data can be read, and suggests:

• Post-processing rules that keep generated values inside the limits of your real data

• Relationships between parent and child tables that the model should keep when it generates related tables

When to run Analyze

Run Analyze after you've set which columns are active, and before you train. It's optional, but quick and worth running every time:

• It's an early check. If Analyze can't read your data, training won't be able to either. You find out in minutes instead of partway through a long run.

• It gives you a starting set of post-processing rules, based on your real data.

• It finds relationships across tables when you're training more than one related table.

Run Analyze

1. Open your data activity.

2. From the Actions panel, choose Analyze.

3. Enter a threshold between 0 and 1, or leave the default of 0.3.

4. Start the job and open the job log. When Analyze finishes, the log lists its suggestions, and the reports are saved on your host server.

The threshold

The threshold only affects relationship suggestions. It doesn't affect post-processing rule suggestions.

It sets how strong the evidence for a relationship must be before Analyze reports it:

Start with the default. If Analyze reports relationships you know aren't real, raise it. If it misses one you know is there, lower it.

Post-processing rule suggestions

Analyze looks at each column and records limits the real data always keeps to, such as the range of a numeric column or whether it only holds whole numbers. It suggests a post-processing rule for each.

Each suggestion appears in the log with a severity, the rule, the table and column(s), and the reason:

[MEDIUM] range_clipping on 'CUSTOMER'.credit_score - source values fall in
[473.0, 850.0] and are non-negative
[HIGH] integer_column_consistency on 'CUSTOMER'.tenure_months, credit_score,
credit_limit - these columns are whole numbers in the source data

The suggestions are also saved on your host server as post_generation_suggestions.json.

Relationship suggestions

When your rule set includes related tables, Analyze follows each foreign key from a parent table to its child table. It looks for parent columns that affect what appears in the child rows. For example, customers in a higher income band tending to take out larger loans.

These matter because each table is generated separately. Without them, a generated loan could be for any amount, whoever the customer is. With them, child rows stay consistent with their parent.

Relationships found appear in the log and in Column Correlations for the connection in the Data Dictionary.

If you run Analyze on one table, the log reports Conditional analysis complete: 0 candidate(s). This is expected. See Why did Analyze find no relationships?

Using the results

Post-processing rule suggestions. You have two choices:

• Use them. Add them to your generation rule set, so generated data stays within the ranges and formats of the real data from the first run.

• Leave them out for the first run. Generate once without them to see what the model produces on its own, then add rules to fix what you find. This tells you more about how well the model learned your data.

Either way, check each suggestion makes sense. A range is only as good as the data it came from. If your training data doesn't cover the full range of real values, the suggested range will be too narrow.

Relationship suggestions. Review each one before you train:

• Keep relationships that reflect how your data really works.

• Delete relationships that are coincidence, or that you don't want the generated data to follow.

Why did Analyze find no relationships?

• The rule set has only one table. Analyze only looks for relationships between parent and child tables. Relationships between columns in the same table are left for the model to learn during training. A result of 0 means Analyze didn't look, not that there aren't any.

• The child table isn't in the rule set. Both sides of the foreign key must be included.

• The foreign key isn't defined. Check the connection in the Data Dictionary under Foreign Keys. If the key isn't declared in the database, define it as a soft key.

• The threshold is too high. Run again with a lower threshold.

Why did Analyze only suggest ranges and whole numbers?

line_item_consistency and parent_child_aggregate only apply to data with line numbers or parent totals. For most single tables, range and whole-number rules are all Analyze can suggest. Add any other rules your data needs by hand.

When to run Analyze again

Run it again if you:

• Add or remove tables from the rule set

• Make columns active or inactive

• Change or reload the source data

You don't need to run it again if you only change engine hyperparameters.

Training Output Files

When training finishes, these files are available to download from the data activity:

Post-Processing Rules Reference

A generative model produces realistic rows, but it has no concept of a rule. It might produce a credit score of 912.7, an invoice whose lines don't add up to its total, or a status that jumps from DRAFT to CLOSED without passing through OPEN.

Post-processing rules fix these problems. They're added to the generation rule set and applied to the generated data after the model has run.

How rules run

• Rules run in the order they're listed. Put a rule that recalculates a value after any rules that fix its inputs. For example, clip a quantity to its range before recalculating a total from it.

• Inactive rules are skipped. Use the Active toggle to switch a rule off without deleting it.

• You don't need to retrain to add, change or remove a rule. Generate again to apply it.

Fields every rule has

Every rule takes a Table* (the table it applies to) and an optional Note, as well as the fields listed below. The exception is sql_alignment, which works across all tables and has no Table field.

Required fields are marked *.

Value constraints

Keep individual values inside legal limits.

range_clipping

Clips a numeric column to a minimum and maximum. Values below the minimum are raised to it; values above the maximum are lowered to it.

Use when: generated values fall outside the real range, such as a negative age.

integer_column_consistency

Rounds decimal values back to whole numbers.

Use when: whole-number columns, such as counts or months, come back with decimals.

sign_consistency

Makes the sign of an amount (positive or negative) match a category column.

Use when: for example, debits must be negative and credits positive.

resource_limit

Caps a usage column at a capacity taken from another column, or at a fixed limit.

Use when: for example, a balance must never exceed the row's credit limit.

Calculations

Make one column equal a calculation from others.

product_rule

Sets a column to one column multiplied by another.

Use when: for example, line total = quantity × unit price.

conditional_ratio

Sets a column to a base column multiplied by a rate. The rate can be fixed, or chosen by the value of another column.

Use when: for example, tax = amount × a rate that depends on the region.

zero_sum

Adjusts a numeric column so it adds up to zero within each group.

Use when: for example, the debits and credits in each journal entry must balance.

delta_smoothing

Limits how much a value can change from one row to the next in a series.

Use when: a time series has unrealistic spikes.

Dates and periods

Keep dates and periods in a sensible order.

date_sequence

Makes sure a later date is at least a set number of days after an earlier one.

Use when: for example, an end date falls before its start date.

sequential_event_gate

Keeps a series of date columns in order.

Use when: three or more dates must happen in order, such as ordered, shipped and delivered.

holiday_date_shifter

Moves dates that fall on a weekend or public holiday to the nearest working day.

Use when: dates such as payment or trading dates can only fall on working days.

temporal_continuity

Links periods together so each period's opening value matches the previous period's closing value.

Use when: for example, monthly figures must follow on from each other.

balance_carry_forward

Sets each period's opening balance to the previous period's closing balance.

Use when: account balances must carry over from one statement to the next. A simpler version of temporal_continuity for balances.

temporal_completeness

Makes sure every entity has a row for every period, and adds rows to fill any gaps.

Use when: for example, every account needs a row for each of 12 months.

Relationships between tables

Keep parent and child tables consistent.

relational_alignment

Makes sure every child row points to a parent that exists. Can remove child rows that don't.

foreign_key_sync

Changes foreign keys that point to a parent that doesn't exist, so they point to a real parent instead.

foreign_key_sync or relational_alignment? foreign_key_sync keeps the row and fixes its link. relational_alignment can remove the row instead.

parent_child_aggregate

Either recalculates a parent value from its child rows, or adjusts the child rows to match the parent.

Use when: for example, an order total must equal the sum of its lines.

cross_table_sync

Sets a parent total to the sum of its child line amounts.

cross_table_sync or parent_child_aggregate? cross_table_sync only sums lines into a parent total. parent_child_aggregate takes any calculation, and can adjust the children instead of the parent.

Structure and grouping

Control how rows are organised.

line_item_consistency

Numbers line items 1, 2, 3 … with no gaps within each group.

stochastic_grouping

Groups rows into documents, each with a random number of lines.

Use when: for example, generated invoice lines need to be grouped into invoices.

state_machine

Makes sure each entity's status only changes in an allowed order.

Use when: for example, an order must go DRAFT → OPEN → CLOSED.

Testing and privacy

Rules that deliberately change data quality, or check it.

outlier_injection_profile

Adds deliberate bad values, so you can test how other systems handle them.

This is the only rule that makes data worse on purpose.

privacy_similarity_score

Finds synthetic rows that are too similar to real rows, and flags or removes them.

Use when: you're using synthetic data for privacy or compliance. A model can sometimes reproduce real rows, and this rule checks for it.

Custom logic

For anything the built-in rules don't cover.

custom_python

Runs Python code against the table's data, held as a DataFrame called df.

The code runs in a restricted environment. df is the only object available. You can use methods on df and its columns, but the pandas module itself (pd.) isn't available and you can't import other modules.

sql_alignment

Runs SQL (DuckDB) across all the generated tables.

Example:

UPDATE JET SET debitCreditIndicator = IF(amountLC >= 0, 'D', 'C');

This is the only rule with no Table field. It can read and change every generated table.

Quick index