The AI passed a test it had already seen: protecting the independence of an evaluation
Separate known regression cases from fresh evaluation material and record how test examples were used.
Article language: English

In this article
A team improves an instruction, runs a test and sees a green result. Difficult examples from the previous version now receive correct answers. That is good news, but what exactly has been demonstrated: repair of known cases, or performance on new ones? Those are different achievements. A green indicator cannot distinguish them by itself.
A test set can lose independence in ordinary ways. Someone copies an extract into the instruction, prepares a demonstration on the same material, or adds an exception after every failed answer. Eventually the test becomes a shared exercise. No intrusion is necessary: a file called “really_final_test” is enough.
A known error can remain in regression tests
Consider a fictional assistant that organises office-equipment requests. It must extract the item, the problem and actions already taken by the requester. In one artificial case, the model invents a check absent from the message. The team corrects the instruction and keeps the case to check whether the problem returns. That is sensible work.
The known example belongs in regression testing, which checks for the return of a previously fixed error. Do not discard it merely because it helped improve the solution. Describe its role: it tests material the team already knows and may have used while making changes. To assess new requests, also prepare material not used to improve the instruction. It should concern the same task and similar working conditions, but contain new situations. Otherwise an effective rule and adaptation to rehearsed examples may be mistaken for the same cause of a good result.
Google's course on dividing datasets separates training, validation and test sets, and describes how repeated use to guide changes can wear out evaluation sets. Its subject is machine learning. In a company's workflow, it is similarly useful to distinguish material for corrections from material for final acceptance.
Adding an example to a prompt changes the task context or the instruction guiding the answer. That does not change model weights. Fine-tuning on examples is a separate learning mechanism. Record what was actually done rather than calling every correction “retraining AI”. Preserve the role of each trial in AI response regression tests (in Polish). Known errors help monitor repairs; fresh material asks about subsequent use. You may need both, with separate results and interpretations.
Trace the journey of one example
Follow a request from the moment an error is found. It might appear in a note, the model instruction, a knowledge base, a report and a presentation for the person approving deployment. Renaming a file does not make an example new. To check independence, you need to know what the solution and the people improving it could have used.
In the working case, record that the first message identified the error and was then added to regression. If its reference answer also entered the instruction, record that too. This helps explain why success on that item is insufficient to accept the whole process.
Do not check only identical text. The same case can appear as a short message, a conversation excerpt and its summary. Shared details may lead to the same decision. Placing those versions in different sets can create an appearance of independence while testing a close relative of a known case.
When Benchmarks Leak, published in January 2026, describes test examples or close variants appearing in model training data. That differs from a company copying material into an instruction. Both situations nevertheless call for care in interpreting a result when test content may already have contributed to preparation.
Identify families of similar cases. If one request was divided into several messages, record their common origin. If a template generates variants of the same situation, retain that fact. The number of rows alone does not tell you how many different problems were posed. Different names of people or objects do not establish novelty either.
You can make a useful control variant by changing a detail that affects the answer. State which condition changed and what the trial is intended to show. Do not automatically present it as a completely independent case from everyday work. If you cannot inspect the base model's preparation data, state that limit. Organising your own files cannot prove that a public benchmark was never known to the provider. You can control the described company process and selection of new material. That is valuable, but narrower than proof of no prior exposure at all.
The information source and answer key have different roles
In a solution answering from company instructions, access to those instructions may be part of the task. There is no need to remove the correct document merely because it will be used to check an answer. Decide what the evaluation should demonstrate: use of an allowed source, or a solution without it. These are different working conditions and need appropriately prepared material.
Separate the test content from its expected result. If the system receives the reference answer with the question, the test may check copying rather than independently combining the evidence. Inspect the entire package sent to the model, including field names and notes that might accidentally contain the answer label or an instruction for choosing it.
For a RAG process (in Polish), record which source collection was available and what was actually retrieved. Retrieval and answering can be checked separately. A missing document will then not be described as inability to use material that was read, and the presence of a correct source will not by itself be called test leakage.
Manage access to reference answers and the status of materials within the team. The person improving the instruction should know which set can be used and which is waiting for acceptance. Do not rely only on meeting participants' memories. A short note about purpose and use history can be more useful than another file marked “secret”.

Fresh material should still resemble the work
Independence does not mean choosing a completely different category merely to prevent repetition. If the assistant organises office-equipment requests, a poetry-analysis test says little about its usefulness for that task. Keep the objective and realistic conditions, while changing the particular situations that did not contribute to the corrections.
Prepare material with different levels of completeness. Some requests may omit the item, some may describe several problems, and others may explicitly deny an earlier suspicion. Reference answers should follow the text, not the team's default habits. Only then can assessment show whether the solution preserves the important meaning.
Set the reference answer before reading the model's result. If you add a correct answer only after its suggestion, you may unconsciously adapt the expectation. When the source is ambiguous, resolve that ambiguity separately or mark the case for further assessment. A tool should not score points for guessing an expectation the team had not written down.
When model suggestions later enter the reference answers, assess feedback from AI's own prompts (in Polish), not just similarity between test files. Keep material current too. A message may refer to an old instruction while you assess against a new one. That can be an intentional version-checking case; if accidental, it makes fair comparison harder. The record should identify which state of information determines the result.
For an AI pilot (in Polish), record how the sample was selected. Otherwise the report reader sees only a final number. They need to know whether it concerns new ordinary requests, prepared exceptions or known cases after repair. Each group can be useful, but answers a different question.
You need not hide the entire method from the provider. Assessment principles and expected behaviour can be public. Separate them from the particular unused examples and answers. Otherwise a secret exam becomes an organisational puzzle instead of a test of clearly agreed work.
Aeneia Audit may be approached to discuss the origin and use of examples, questions for assessing a tool and limits of conclusions from trials. The scope and availability of any work require agreement.
A change after the result calls for another trial
Imagine a fresh request reveals a new error and the team immediately improves the instruction. That may be the right response. The particular case has now become part of work on the solution. Keep the history, the result before the change and the check of the correction. For the next assessment on new material, you need another case not previously used.
A set put aside can also enter work on a later version. Knowledge moves between people and requirements change after discussions. An explicit status and refresh plan are more useful. The reader can then see which material still serves as an acceptance trial and which now supports improvement.
When changing the reference answer, record why. Correcting an obvious labelling mistake differs from expanding requirements after seeing a good answer. In the latter case earlier results may no longer be comparable. Change history lets you return to the original question instead of moving the goal after every attempt.
Treat repeated runs separately from data freshness. Response repeatability (in Polish) measures variation on a defined task. Even very stable behaviour does not confirm that the material remained independent. Conversely, new cases do not remove the need to observe variation when it matters for use.
If the model should sometimes decline to answer, include that in the reference. A good refusal when evidence is missing (in Polish) requires distinguishing incomplete input from an execution error. Pressure for a complete answer every time may reward invented details. An independent test with a faulty criterion is still a faulty test.
The report should show where confidence came from
Connect the result to the solution version, the set's role and the date of use. The reader should be able to distinguish what was checked for the first time from what was repaired and tested again. They do not need every working note on page one, but they need a route to the material if they want to check the basis of a decision.
If two versions were compared on different trials, do not present the score difference as a pure change in quality. The new material may have been harder or covered different situations. Describe both results and their scope. Direct comparison requires conditions that allow the effect of this difference to be understood.
Connect conclusions to the experiment hypothesis card (in Polish). If the question was whether the solution handles new requests, show evidence about new cases. Successfully repairing several known answers is useful work, but does not automatically settle that question.
For a conversation with Aeneia Audit, prepare the history of one example: its origin, who saw it, what was changed using it and when the result was considered correct. That trail helps show whether the test still examines the solution or the team has been testing familiarity with its own test.









