Skip to content
Interface language: English
Audit / Signal

AI says the tests passed. Could they have caught the bug?

Assess AI test reports through the command, assertions, results and a controlled defect on an isolated copy.

Article language: English

A stack of paper tied with a dark ribbon and a gold-coloured seal rests beside a pen.
In this article

“Twelve tests, all passed.” A coding assistant's report can be reassuring, particularly after an hour spent discussing a fix. Before closing the task, one more question is useful: which of those tests would stop passing if the fix were wrong?

The case name, the response being checked and the outcome of the run help distinguish evidence about a particular behaviour from evidence that a command was executed. When working with AI, the two can easily merge into a brief “done”. It turns out that “done” can cover a great deal.

A report starts with what actually ran

Ask for the exact command, where it was run and the tool's result. A script name is not enough. In one project, a command named test may run the relevant suite; in another, it may only check formatting or print a help message. The person accepting the work should be able to establish the scope without guessing what the person who named it intended.

Check how many tests were collected, run, skipped and failed. The end of a process alone does not say whether it included a case for the fix. If a report lists twelve cases, it should be possible to identify the one that checks the new behaviour. The total can stay the same even when an entirely different directory was run.

The pytest exit-code documentation assigns code 0 to tests collected and passed successfully, and code 5 to no tests collected. This distinction belongs to that particular tool. A green result from a wrapper script does not replace information about what pytest itself returned and which tests it found.

Distinguish a run from a plan to run. An assistant may write a correct command it has not executed, or describe a test whose file it is only proposing to add. Both can be useful, but neither confirms a result. If the environment blocked execution, preserve that limitation in the final answer.

For a long log, retain the final result and the complete record needed to see the scope. Truncated tool output may contain the first passing cases while leaving out a later failure. The assistant should read the end of the process before calling the test successful. A command that is still running is not a positive result either; it means the outcome is pending.

Establish which version was used as well. A test from before the last edit does not automatically approve a file after that edit. The saved code version, test version and execution log should form one identifiable set. For local changes, you can preserve an exact copy of the files being assessed; a label for an earlier saved version does not necessarily include current local changes.

If a test starts another program or service, check what counts as its result. “Request sent” may be only the beginning of the operation. When assessing your own test environment, connect the run to its observed effect. The article on AI automation test environments (in Polish) discusses how to set up such a test. The same order applies to code tests: performing an action and checking its result are separate parts.

An assertion must recognise the right result

An assertion checks whether an observed result matches an expectation. In our hypothetical registration system, a workshop has a limit of six participants. The assistant's task is to allow six people to register and refuse seven. The test should distinguish those situations according to the rule agreed for the workshop.

Calling a function without checking its response may detect a crash, but it need not detect a wrong decision. The program could politely approve any number of places. The test would still finish calmly if it expected only that execution would not fail. Write down what success means: accepting six and refusing seven, not merely calling the function.

The expectation should not be supplied by the same piece of program being checked. If a test asks the implementation for the correct answer and then compares it with that same answer, it may only confirm that the implementation agrees with itself. The rule “six allowed, seven refused” should come from the requirement and be visible in the test case.

If the assistant changed the test too, examine the difference in expectations. Changing “refuse seven” to “accept seven” can make faulty code pass every case. It may also be justified by a changed requirement, but then that decision needs to be named and approved separately. A green result after changing both the program and the criterion cannot, on its own, establish what was fixed.

For a reported defect, ask for a case that reproduces its particular symptom. If it can be run on the version before the fix, it should show the mismatch with the requirement there and pass after the fix. This does not mean building a large test history for every cosmetic change. It does help establish whether the added test concerns the problem it was supposed to address.

Check whether the data can distinguish the correct version from the faulty one. A case with two people will not expose a change from “at most six” to “fewer than six”. Two people fit both versions. A case exactly on the boundary tests that distinction without running a hundred random numbers.

Not every requirement is a numerical rule, however. If the fix concerns the message after a refusal, an appropriate test might check that the reason is shown and the entered data is retained. If it concerns saving, it may be necessary to check that the data can actually be read later. Choose evidence for the behaviour being accepted instead of demanding the same form of test everywhere.

A test involving an external service may use a mock, a controlled substitute for the real connection. It can check selected program responses. It does not prove that the real service works. If the mock always returns success, a missing response or a refusal remains outside that test's scope. The report should name which part was checked using a substitute.

Inspect error handling in the test itself. If it catches every error and finishes without checking, it may report success even when the function under test never returned a response. Distinguish an expected registration refusal from a connection failure and an error in test setup. All can interrupt the normal path, but they prove different things.

Also consider whether an expectation is too general to answer the question. “A response exists” does not establish whether it refuses seven people. “A list was returned” does not confirm the right participants. An assistant can write a syntactically correct assertion that checks a property too weak for the stated goal.

Aeneia conceptual photography

A controlled defect reveals what a test can detect

In its introduction to mutation testing, Stryker describes deliberately changing code and running tests again. The expected test failure shows that the particular change was detected. This way of thinking helps assess the strength of a specific case even if you do not run a full mutation-testing tool.

Code coverage, information about which parts ran during tests, does not by itself answer whether their results were correct. Stryker's authors point out that distinction. When accepting our fix, identify the rule being checked and check the result that should change when that rule is broken.

In a copy of the workshop example, you can deliberately change the condition so that six people are refused. The boundary test selected earlier should now fail for that behaviour. If it still passes, establish whether the right test ran and whether it actually used the changed function.

Change one thing at a time and describe the expected signal. If you break the entire file so it cannot be loaded, you mainly learn whether the loading error is detected. That is different evidence from recognising the wrong decision about the number of places. The defect should be precise enough for the outcome to answer the question about the rule under test.

Run this check on an isolated copy and restore the correct version. You do not need to introduce bugs into a system in use to assess a test. An AI agent sandbox (in Polish), an environment with a limited scope of action, requires checks on file and network access. Calling a directory “test” does not create that boundary by itself.

After restoring the code, run the same case once more. The sequence is then clear: the correct version passes, the controlled faulty version fails, and the restored version passes again. Save the results alongside the file difference. Do not present this sequence as performed if you only described how it could be performed.

Choose the scope of deliberate defects according to the significance of the change. Select rules where it matters: the capacity boundary, omitted data or a refusal incorrectly treated as success. The aim is an answer to a specific acceptance question, not an impressive count of additional runs.

Aeneia Audit

Let's establish what evidence should support a coding assistant's work. We can assess reporting, control cases and the limits of checking so the team knows what follows from the claim “the tests passed”. Discuss evidence for accepting an AI assistant's work

Tests can pass without reaching the changed code

When a deliberate defect does not change the result, do not immediately alter the expected answer. First check the execution path. The test may have loaded a different copy of the file, an earlier build or a mock replacing the function. Adding more data to the same test will not check code it never reaches.

Ask which file and function were used. With several similar directories, the function name alone may not be enough. If the application must be built or prepared before the test, check that this was done after the fix. The evidence should connect the current file with the version that ran, not merely place their names next to each other in a report.

Exclusions matter too. If the assistant could not access some files, its check may cover only the visible part. Assess separately whether the restriction was effective and what scope of testing was then possible. These questions are explored in excluded files and a coding assistant's actual access (in Polish).

For an inconsistent result, preserve earlier failures as well. If the same case passes sometimes and fails at other times, one successful run does not explain the cause. Check for a dependency on the current time, execution order or data left by a previous test. Once the identified cause has been removed, agree on a confirming run. Repeating until green does not by itself remove uncertainty.

Do not count a skipped test as one that was run and passed. The reason for skipping may be legitimate, such as lack of access to a necessary service. The question still remains unchecked in that environment. In the final assessment, record the missing evidence and how to obtain it instead of putting everything into one green number.

If a run stopped because a library was missing, establish the dependency's origin first. Checking packages proposed by AI (in Polish) concerns a different problem from assertion strength. Adding a random package just to remove a red message settles neither question.

The final report should support acceptance of a specific change

Instead of “everything works”, record what behaviour was checked, in which version and with what result. In our example, a report could identify acceptance of six people, refusal of seven and a passing case after restoring the correct condition. If the controlled defect was not introduced, describe only the checks actually performed.

Accepting code and the work of AI agents (in Polish) should connect such evidence with responsibility for further use. The person accepting the change needs to know the remaining limits: mocks, unavailable services, skipped cases and review scope. They do not need the history of every unsuccessful command unless it explains the current result.

Aeneia Audit can assess how AI is used, the acceptance criteria and the quality of reported evidence. A specialist review of the code itself requires appropriate expertise and an agreed scope. Here, the acceptance question is whether the assistant's claim can be connected to a specific test that ran and what it means.

After such a report, “twelve tests” can remain a useful shorthand. Behind it, however, there should be cases capable of challenging a faulty change. Green should reassure us because something was checked, not because it looks good beside the word “done”.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear