Skip to content
Interface language: English
Audit / Signal

AI answer quality after a change: how to run regression tests

Check whether changes to models, instructions or sources have weakened an AI workflow. Keep a baseline, classify errors and define acceptance and rollback conditions.

Article language: English

A gardener adjusts the water flow at a prototype made from roots beneath a red greenhouse canopy.
In this article

Yesterday, the process recognised documents correctly. Today, after an update, it mixes up two categories. A regression test is meant to catch that kind of deterioration: it compares the new behaviour with the behaviour the team already relied on. Keep the cases, expected results and test conditions. Check the whole task, including refusal and handing a case over to a person. A new model number alone does not tell you whether its answers are reliable.

An initial pilot asks whether a solution is suitable for a particular use. Later comparisons serve a different purpose: to establish what has changed from the approved way of working. They therefore need a preserved baseline and repeatable cases. One good answer after an update tells you little about how the system will handle the other situations.

Record exactly what you are changing

Create a change record that identifies the component, reason, owner and expected effect. Distinguish changes to the model, instructions, settings, source material and permissions. If you update several components at once, note that identifying the cause of a later difference may be harder. Where feasible, assess successive changes separately and keep their results.

Record the model or service identifier, instruction version, sources and test date in the baseline. If the provider does not expose a detailed version, keep the information available and state the limitation. It is worth agreeing how updates can be tracked during the AI vendor assessment. Otherwise, a user complaint may be the company's first notice that something has changed. For more context, see the article on assessing an AI vendor before purchase (in Polish).

Keep a set of cases from the working process

Choose material that represents routine work, important exceptions and previously identified errors. Include straightforward cases as well as situations where the correct result is a follow-up question, a refusal or an explicit statement that information is missing. Record where the data came from and how it may be used. Quality testing does not justify copying arbitrary documents into another environment.

Do not limit the set to examples used to refine the instructions. Keep some cases exclusively for evaluation. If the team knows them by heart and continually adapts the solution to them, the test covers an increasingly narrow capability. Expand the set as new situations arise, recording when each case became part of the comparison.

Assess the qualities of the result, not identical wording

Many tasks allow several correct answers. Define the information that must be present, the acceptable room for variation and the errors you cannot accept. In a summary, findings and agreements, conditions, names and whether they are supported by the source may matter. A different sentence order does not necessarily mean a regression. Adding a decision that was never made requires a separate assessment of its consequences.

For each criterion, specify how you will check it. Some checks can be automated, such as whether a required field is present. Others require someone who understands the task to compare the answer with the document. A format check cannot replace a check of the substance. A correctly structured answer can still contain incorrect findings. The article on criteria for evaluating an AI pilot (in Polish) revisits the criteria for the initial task.

Classify errors by their consequences

Define the categories before making the comparison. A process-critical error might involve a prohibited action or disclosure of material beyond the permitted scope. A significant factual error needs correction before acceptance. A minor stylistic flaw may not prevent use. Choose category names and boundaries to suit your task; no single universal scale fits every company.

Agree how to handle an answer that is correct but takes much longer to check. Process quality also includes the effort required of the recipient and how clearly uncertainty is expressed. Record the effect of such a change on the total cost of working with AI (in Polish). This prevents an improvement in one measure from hiding a deterioration in another important condition.

Compare under the same conditions

Run the old and new variants on the same version of the material, with comparable permissions and settings. Distinguish raw results from results corrected by a person. If the earlier output was manually refined, preserve that information. Comparing it with a new raw answer could make extra human work look like a difference between the variants.

Also record technical failures and timeouts. Do not omit failed attempts from the run summary simply because they produced no text. If a provider has changed a feature so that the old variant can no longer be run, use the saved results and explain the limitation. The decision should show what was actually compared and which parts of the process had to be assessed another way.

Allow for variation between answers

The same material can produce different results across runs. Choose the number of repetitions according to the consequences of an error, the variability of the task and the resources available. Do not present one successful answer as proof of consistent reliability. Keep both the result and the test conditions so that the person responsible for acceptance can distinguish a single observation from recurring behaviour.

The official OpenAI evaluation guide recommends task-specific tests, evaluation after changes and combining metrics with human judgment. It is a useful methodological reference. Acceptance thresholds, the number of trials and the acceptable error level remain decisions for the particular process, rather than values to copy directly from a provider's example.

Aeneia conceptual photography

Test sources and access boundaries

A change in quality may come from retrieving different material even when the model itself behaves similarly. Test a current document, a withdrawn version, a missing source and conflicting material. Assess whether the answer identifies its basis and avoids filling gaps with invented content. If sources have dates and owners, keep that information with the test case.

Include a test using an account that has no access to the selected resource. The expected refusal is part of correct behaviour. Also check what happens after permission is revoked, particularly when the product maintains its own index or copy. The guide to reviewing AI integration permissions (in Polish) explains these checks in more detail, while the article on organising a company knowledge base (in Polish) covers organising the material.

Align human and automated assessment

If another model provides an initial assessment, check its agreement with people who understand the task. Keep a sample of disagreements and agree how they will be resolved. Automated assessment can speed up a review, but it is itself a component of the solution that needs checking. Changing the evaluator model can change the assessment result even when the assessed system's answers remain identical. A separate guide to reviewing an AI model used as a judge (in Polish) helps define these checks before its verdict becomes the basis for an update decision.

For human assessment, prepare a short guide with examples of errors. Ask two people to review a selected part of the set independently. If they interpret a criterion differently, clarify the rule and record what changed and when. The article on training a team to verify AI answers (in Polish) shows how to practise moving from an impression of correctness to identifying evidence.

Worked example: changing a summarisation model

A team has an approved process for producing meeting notes. After a model change, the answers are shorter and easier to read, but some omit a condition on which carrying out the task depends. In the comparison set, that condition is mandatory. The reviewer records its omission as a substantive error, regardless of the improvement in style.

The process owner can then pause the change, restrict it to a different type of material or refine the solution and evaluate it again. If the instructions are improved using the revealed examples, new material is also checked. The result should not rest solely on cases used to make the correction. This is a fictional training example of how to make the decision, without attributing failures or results to a real client.

Define rollout and rollback conditions

Before launching the new variant, record the conditions for proceeding, the person who approves it and how it will be monitored after the change. Sometimes you can start with a limited group of tasks. In that case, specify the boundaries, trial period and criterion for expansion. A bare “tests passed” status, without the set and its limitations, is not a sufficient basis for handing the process to another team. If a coding assistant prepares or runs some of the tests, check the command, assertion and actual test result (in Polish).

Check whether returning to the previous variant is technically possible. If it is not, prepare an alternative, such as manual execution or reduced functionality. Record who can stop automated actions and how to identify tasks that need checking again. A guide to auditing process automation (in Polish), covering queues and work in progress as well, can help review the whole flow.

Keep a reproducible record of the result

The acceptance package should include the change record, test-set version, criteria, raw answers, assessments, disagreements and decision. Restrict access to the material according to its contents. Set a retention period and assign an owner for updates. The next reviewer needs to know why the variant was approved, not just see a favourable number in a report.

Show improved, worsened and unresolved cases in the report. A total score should not hide a significant error or a different effect on a subset of tasks. Add a list of uses that remain outside the assessment. If the test covered summaries, do not treat it as confirmation of message sending, analysis of other documents or autonomous agent actions. Each extension needs an appropriate basis.

When should the test set itself change?

Review the set when the users, document types or use of the answers change. An old sample may still compare versions well while becoming less representative of current work. Keep a common subset to track continuity and add new cases that reflect the current scope. The report should distinguish results for the two groups.

Remove or correct a case if its source has become outdated or the reference answer contains an error. Keep the reason and date of that decision. Do not, however, use an inconvenient result alone as a reason to withdraw a test. First establish whether it still reflects a real requirement. Otherwise, it is easy to produce an apparent improvement by changing the question instead of improving the solution. When accepting a new set, ask the process owner to confirm which decisions it can still support.

For a larger update, also use the comparison before changing an AI model. Keep the same cases and separate the effect of the model from changes to instructions or sources. Replacing every component at once makes it harder to explain where the new result came from. Further guidance is available in the article on comparing results when changing an AI model (in Polish).

How can an established team start?

Choose the next planned change and a small, justified set of cases from current work. Record the baseline result, criteria and responsibility for acceptance. If no earlier data exists, call this stage the creation of a baseline. The next comparison will then have a real point of reference. You do not need to wait for an elaborate testing platform to put the decision on a sound footing.

The Audit service overview shows how to choose the scope of a review.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear