AI-generated synthetic data: every row looks right, but the whole set does not add up
Check AI-generated test data beyond individual rows: examine record links, event histories and the range of cases the whole set is meant to cover.
Article language: English

In this article
A test rental company has excellent data. Every stand has a name, every reservation a number, and every date is written correctly. Only after the files are combined does the team discover that one stand was returned a day before it was handed over, while another was rented to three people at once. The rows behave impeccably. Trouble starts when they meet.
This is a fictional example of a set generated to test a process. Synthetic data, created instead of copying real records directly, can look convincing without forming a useful test world. Examining a few attractive records will not show whether the links, sequence and number of cases fit the task.
Before generating anything, write down what the set is supposed to test. A catalogue demonstration, a check that conflicting reservations are blocked, and an assessment of a model trained on examples each need different data. Do not use the single label “good test data” as a shared judgement for all three.
First describe the world in which the records can exist
Our rental company has equipment types, individual items, reservations, and handover and return events. A type of stand is not the same thing as a particular stand with an inventory number. A reservation for ten items needs a different link from a reservation for one individual item. Set out these meanings before asking AI to prepare a hundred realistic entries.
For each object, specify an identifier, a value that distinguishes that exact record. Then record the permitted relationships. An item belongs to an existing equipment type. A reservation line points to the right item or to the agreed quantity of a given type. A return event refers to a specific handover, not merely to the surname of someone who once rented something similar.
Keep the meaning of identifiers consistent across files. “Egzemplarz01” generated independently for two warehouses may refer to two different physical items. Once the packages are combined, the application will see one ambiguous number. Decide whether identification also needs a warehouse code, and check the whole agreed combination. Do not fix the collision by changing the number only in the catalogue: references in reservations and events must still point to the same thing. Each part may look correct on its own, with the conflict appearing only when the set is combined.
Start with a small world checked by hand. A few equipment types and individual items are enough to expose ambiguity in the schema. Increase the number of records only afterwards. If you cannot establish the links among six records, a thousand AI-generated rows are unlikely to supply the process rules for you.
Rules can apply to a single field, a whole record or an entire set. A positive quantity is a field condition. A return date later than the handover date is a relationship between times. Preventing two active reservations of the same item for conflicting periods requires comparing different records. A valid date format alone does not cover the other levels.
Separate a rule from a convenient assumption. Does the rental company allow partial returns? Can one delivery serve several reservations? Can a planned reservation be cancelled before handover? The process owner should answer these questions. Do not forbid the generator from creating events that really are possible merely because they make the first example harder.
A check of AI-generated JSON data (in Polish) examines, among other things, what values in a record mean. Here the whole set matters too: a valid line can point to the wrong object or break a timing rule only when compared with another line.
An existing identifier is only the start of a relationship
Collect all references and check them against the right sets. An item number in a reservation should appear in the item list. A name resembling an existing one is not enough. If two stand versions differ only in the end of their codes, automatic matching on the word “stand” can connect a reservation to the wrong item.
SDMetrics describes ReferentialIntegrity as checking whether a foreign key can be found in a primary key, or whether a reference matches an identifier. This metric counts missing values as valid foreign keys. A good score therefore does not replace a separate rule requiring an object to be specified.
For the set, state explicitly whether a reservation line may not yet have an item assigned. At an early stage, perhaps only the equipment type is reserved and the individual item is chosen later. If so, a missing identifier represents a particular process state. When handling a handover, that same absence may prevent the action. The rule depends on the stage, not just the empty cell.
Also check whether the item belongs to the type selected in the reservation. Both identifiers can exist while the relationship is still wrong. It is like a valid room number that belongs to another building. In the report, show the specific broken dependency and the records needed to reproduce it.
A different error occurs when the generator reverses a relationship. “The new reservation replaces the previous one” does not mean the same as “the previous reservation replaces the new one”. The article on direction and evidence for relationships in an AI knowledge graph (in Polish) explores similar distinctions. In test data, the name of a connection should match the actual action the application is meant to perform.

Use the set in the queries needed for the test. When asking for the number of reservations, joining them to multiple lines may duplicate rows. A small example of your own with a known result helps determine whether the error comes from the data or the query. The guide on row counts and case counts in AI-generated SQL (in Polish) illustrates the difference. A test should not pass merely because the same generator prepared both the package and the expected result.
The right sequence of events needs a history
For one reservation, show a sequence: creation, optional confirmation, handover, return and closure. Do not impose it on every state without exception. A cancelled reservation may never reach handover, and a partial return need not close the whole reservation. Record the allowed transitions and the conditions that distinguish these paths.
Check the time an event happened separately from the time it was recorded. A staff member may record a return after actually receiving the equipment. That later entry is not the same as a return before handover. If the schema contains both times, the generator should preserve their meanings. If there is only one field, agree which moment it describes.
Assess conflicts according to the agreed availability rules. May periods meet at their boundaries? Is preparation time required between return and the next handover? The process owner must decide. There is no single magic rule that “dates do not overlap” which describes every rental without these conditions.
A control set should contain both a valid history and a deliberately invalid one. List an error introduced to check whether it is detected in a separate set of expectations. Do not repair it automatically while preparing the data.
For a clear test, start with one valid history and change one dependency at a time. In one copy, replace the item number with one that does not exist. In another, keep the numbers and move the actual return before the handover. In the next, introduce a conflicting reservation for the same item. For each, write down what should be detected and what should remain correct. A set with five independent faults may be useful later, but at first it makes it harder to explain which check actually worked. This is a test plan, not an account of running an application.
Keep a package of valid examples separate from one intended to test rejection. For the latter, identify the broken rule and the expected type of response. A high share of rejected records may be a successful negative test, but a failure for a generator promising a valid world. Without naming the set's purpose, the same result can be presented as either success or failure.
A valid world and a useful test are different judgements
Imagine a thousand valid reservations with exactly one line each. Every reference works, the dates are in order and the application passes the test. You still do not know what it will do with a reservation containing several lines, a partial return or a type of equipment that nobody has rented yet. A large set can repeat a narrow case very precisely.
SDMetrics' CardinalityShapeSimilarity compares the distribution of child rows per parent row in real and synthetic data. It therefore assesses a different property from whether references between tables exist.
For our test, count, for example, how many reservations have one, several or many lines. Check whether there are items with no history and items used repeatedly in successive periods. You do not have to reproduce real proportions when the aim is to test a rare event. You do need to state whether the set should resemble ordinary incoming activity or deliberately concentrate difficult cases.
Distinguish variety in names from variety in histories. A hundred reservations for a hundred fictional people may repeat the same path: one item, a period and a full return. Changing names does not add a test of partial handover, a change of date or cancellation of one line. At acceptance, show the required situations and identify the records that realise each one. If a case is missing, add it deliberately rather than generating a larger package again in the hope that the needed history will happen to appear.
Assessing an AI model fine-tuning dataset (in Polish) raises different questions about selecting examples and a later independent test. Do not automatically transfer an application test result to training material. A package ideal for demonstrating three screens may lack the variety needed to assess how a solution behaves on other histories.
The label “synthetic” does not by itself establish anonymity. Find out where the material used by the generator came from and what may be shared further. If the aim is to test reservation relationships, entirely fictional names and contact details are often enough. Do not add real data to an attractive preview just because the other fields were generated.
Run tests in your own test environment, keeping fictional events distinct from current operations. A guide to AI automation test environments (in Polish) discusses preparing such a test. An email address in test data should not accidentally trigger a real notification to a customer. A consistent package and a safe place to use it are separate acceptance conditions.
Keep the generator or prompt version, the schema, rules, the set itself and the results of checking it. After correcting a dependency, repeat the relevant tests. Manually removing a few troublesome rows can change the history of other objects. If you delete a handover, a remaining return loses its basis. If you remove a conflicting reservation, also check its associated lines and events. A correction should cover the identified part of the history, followed by another check of its dependencies. Do not judge the repaired package solely by the disappearance of the first error message. The team should know which package was actually used to confirm the behaviour.
For an Aeneia Audit review, prepare a small set and a description of one process it is meant to support. Choose a valid history, a missing link and an event in the wrong order. These provide a concrete starting point for assessing the material's quality. A good-looking record is a good start. Before acceptance, let it meet its neighbours.











