Valid JSON, wrong data: what an AI response schema cannot check for you
Valid JSON is not enough. Check units, missing values, identifiers and agreement with the source and process.
Article language: English

In this article
The result is green. The parser accepts the JSON, every field has the right type and the application receives the answer without an error. Yet the record says a panel is 240 metres long instead of 240 centimetres. Formally everything checks out. The warehouse would just need slightly larger doors.
Structured data helps pass AI results to applications by limiting format freedom. It does not replace checking whether values mean what they should. An audit should separate valid syntax, schema compliance, and agreement with the document and task rules. A green result at one stage should not colour the others.
Follow a fictional process in which a model reads an equipment sheet and prepares a record for an order system. It needs a product identifier, length, unit, quantity and availability date. The example is small enough to check each step while revealing several different error classes.
What does the schema actually promise?
JSON is a way to represent data in a recognisable structure: an object contains named fields that hold values. A schema sets requirements for that representation, such as a numeric type or the presence of an identifier. A non-technical person need not know all the syntax to ask exactly what is checked and what validation does not cover.
JSON Schema's object documentation explains that listing fields under properties does not make them mandatory. required serves that purpose, while additional fields have separate rules. Assess the schema actually used rather than a general claim that "the answer is JSON".
In our process, the length field may require a number. Both 2.4 and 240 are numbers, though they describe different values. A schema can restrict some allowed data if the relevant conditions are written and the mechanism supports them. It will not determine by itself which passage on the source sheet is the length of the selected variant.
Anthropic describes structured outputs as constraining responses to a supported schema and lists limits on that support, including numeric constraints. Do not assume every rule in any schema is automatically enforced by every tool. Check the specific contract of the configuration in use.
Keep a copy and version of the schema in use. If the instructions say centimetres, the schema says metres and the screen says millimetres, the model is not the only place an error can arise. Review all three descriptions with someone who knows the process. Resolve the differences before comparing responses so the test has one agreed expectation.
For acceptance, prepare a short map: what the parser rejects, what structural validation checks, what is compared with the source and what needs a process rule. Assign each field to the responsible stage. This is easier for a team to review than "the model should return correct data", which sounds reasonable but does not say where an error will be caught.
A number needs meaning
The product sheet gives a length of 240 cm. The agreed system expects metres. The correct result is 2.4 m, but a model might copy 240 and add the unit expected by the application. Each field looks valid on its own. The relationship between value, unit and source is wrong.
In the test, retain both the source value and the converted result. Record the expected rule and assess field selection and conversion separately. If width was read instead of length, a correct conversion will not repair the record. Do not put both mistakes in a generic "number problem" category in the report.
Add two variants and similar values to the material. A sheet may show product dimensions and packaging dimensions. The system should return data appropriate to the assigned task. The presence of a correct number somewhere in the document does not prove the answer uses it in the right place.
For OCR before AI processing (in Polish), also check decimal separators, minus signs and unit labels. An error may occur before the model answers. Retaining the source passage helps distinguish faulty recognition from a wrong interpretation of a correctly recognised value.
Check rounding at an allowed boundary. In the fictional task, choose the rule required by the process and prepare a value just below and one just above the boundary. See whether the model, application and screen show the same result. If a later operation uses the full value while a person approves a rounded one, record that difference as part of acceptance.
Quantity needs a separate assessment. "Two sets of three items" can mean two packages or six items, depending on what the field stores. Do not decide intuitively after seeing the result. The order-unit definition should exist before the process runs and be understandable to people using the record.
In a spreadsheet used in an AI process (in Polish), that difference may feed later calculations. The test should follow the data to where it is used. Correctly summing an incorrectly defined quantity produces an equally wrong result, just with a very carefully calculated total.
A missing value must not pretend to be information
The document has no availability date. The model returns empty text, zero or today's date. Each can pass some superficial checks, yet each means something different to the downstream application. You need an agreed representation of absence and the action that follows from it.

Distinguish a missing field, empty text and null if the contract includes them. Then check how the target system interprets each. Follow the value all the way through: an application can turn empty text into a default even if the model added nothing. The report should identify the stage where that conversion occurred.
Add an ambiguous date case. A digits-only date can have different interpretations if the input format was not agreed. Separately prepare a syntactically valid date that does not fit the task, such as a promotion end date entered as availability. Checking the shape of the date alone will not distinguish those meanings.
In AI calendar planning (in Polish), time zones and the scope of a particular event also matter. The example shows why a field name such as date can be too slight a description of a contract. You need to know whether it is a day, an instant, a deadline or the start of a task.
Add conflicting sources: a table with one date and a note with another. In the expected result, specify whether the process applies an explicit precedence rule or reports a conflict. Do not accept any date merely because it appears in the material. This distinguishes reading a value from deciding which one may be used.
Do not force a model to fill a required field by guessing when the material does not contain it. You can require an explicit missing status or stop the record before use. The choice depends on the process but should be anticipated in the test. A filled table is not the goal if completeness was achieved at the expense of truth.
An identifier can look perfect and point to nothing
The output contains a product code in the expected format. You still need to establish whether that product exists and matches the right variant. Prepare three cases: a known identifier, a nonexistent code with valid structure and an existing code for another variant. Only the first meets the task, although all three can look plausible.
Evaluate normalising the notation separately from correcting its content. Removing spaces from an identifier may be an agreed operation, but changing a similar-looking letter to a digit requires a basis. Keep the input value and transformation rule. Without them, it is hard to establish whether a wrong code arose during recognition, the model's response or the application's automatic correction.
The existence check should use the appropriate catalogue or another agreed source. Do not ask the same model only to confirm that its own proposal is correct. Show the basis and source version in the report. A record in an old catalogue copy may not be enough for current use.
A similar trap occurs with dependencies suggested in AI-generated code (in Polish). A library name can resemble real names without describing the right package. A valid string is only the input to an object-identity check.
When data is meant to trigger an action, connect validation with AI integration permissions (in Polish). Receiving a valid record does not itself authorise an order or catalogue change. In a controlled environment, check what the application does with a rejected record and whether it passes it onwards by another route.
A review with Aeneia Audit can start with one sample input, the response schema and a description of the next operation. The assessment then concerns the actual use instead of ending with a check of brackets.
Review three separate results
In the test record, show structural validity, agreement of values with the source and compliance with process rules. If a record passes the first stage but fails the second, retain that information. It lets you repair the right part without weakening a working format check.
Add a record with correct values assigned to a different document. If preparing a synthetic test set, check consistency of relationships and events in synthetic data (in Polish) so the reference itself does not contain an accidental error. The process should also check the response's relationship to the input. Otherwise every field can pass local checks while the data lands on the wrong case. Prepare two similar documents and deliberately swap responses in a controlled test without writing them to the real system.
Record the versions of the instructions, schema and validation rules. When a record is later corrected, mark whether the input, interpretation or only the representation changed. This lets you repeat the case and investigate why it improved. A newly correct answer alone does not say which stage behaved differently or whether the previous error might return.
Test the rejection response too. Can the operator see the field, reason and passage needed for correction? Does the correction retain its link to the source? A generic "data error" makes a person reread the whole document and hides the cost of using automation.
In a controlled test, also provide an empty response, an interrupted response and one containing only a message that the task cannot be completed. Check whether the application distinguishes these states from a ready record and does not fill required fields with arbitrary values. The process needs a separate ending for an uncompleted task rather than creating an importable object at any cost. Show in the report what the person waiting for the result sees and how they can resume work.
Check a partly correct record. If one field is wrong, decide whether the rest may be used or the whole object must be stopped. The answer depends on the task: a description draft and an automatic order need different treatment. Record the decision in process rules so the technical ease of import does not accidentally become a business rule.
Keep difficult cases for another check after changing the instruction, schema or document-reading method. A small set is enough if it reflects real distinctions: length and packaging, missing and zero, the correct code and a convincing neighbour. A green parser remains useful. It simply no longer has to pretend to be the whole quality-control department on its own.











