AI summarises an A/B test: more clicks do not answer the question
How to check an AI-written A/B report: people and visits, percentages and percentage points, sample allocation and the limits of a claim that one variant is better.
Article language: English

In this article
Variant A produced 50 submitted forms, and variant B produced 60. An AI wrote: "B increased effectiveness by 20%; we recommend rollout." The sentence fits on a slide and looks like a ready decision. It leaves out one detail: A had 1,000 visits and B had 1,500. The denominator did not make it onto the podium.
This is our own hypothetical example. Suppose each form was submitted during a separate visit. In A, 5% of visits ended in a submission; in B, 4% did. B has 20% more submissions, but its rate per visit is one percentage point lower. The arithmetic alone changes the headline. We still do not know whether visits are the right unit for this experiment.
Reviewing an AI-prepared A/B report begins with reconstructing the comparison: what was changed, who was assigned a variant, what was counted and when the trial ended. Separate the report's agreement with the data from a formal assessment of the experiment. Statistical analysis requires a suitable design and expertise; the figures below do not prove that a real form works.
Find the question from before the test
Suppose a company compares two ways to present a "project description" field. In A, a customer sees an empty text area; in B, short guiding questions. The aim is to receive an enquiry with the information needed to prepare for a conversation. The number of clicks on the submit button alone does not establish that the aim was met.
Ask for the hypothesis and outcome definition recorded before the experiment. If the team chose the number of useful enquiries, the model should not replace it with form openings simply because that chart is rising. A business experiment hypothesis card (in Polish) defines the question and assessment conditions before data are collected for precisely this reason.
Check that the report concerns the interface version that was planned. If a required field or the submission behaviour changed during the trial, the material contains more than one comparison. Record the time windows and versions. Combining them without explanation under "variant B" can hide the difference between the first and the corrected version.
Also name the event counted as success. Clicking the button, successful receipt of the form by the server and classification of the enquiry as useful are three successive events. Their counts may differ. A reader should know which one lies behind the word "conversion".
Check when the event is recorded. In the fictional form, pressing the button might trigger a validation error and prompt another attempt. Two clicks do not mean two accepted enquiries. Conversely, a successful submission might not show a confirmation on screen. Choose several such cases for the check and compare the record with the actual outcome. An event name in a dashboard does not prove it is recorded at the right moment.
Agree on a classification rule for enquiry quality. In the fictional example, let a useful enquiry contain the project type and an approximate scope. An assessor should apply the same conditions to both variants. Do not change the definition after noticing that B more often collects long but unhelpful descriptions.
Find the stopping rule and the time the trial actually stopped. Was it to run until a date, an agreed data target or a condition specified by the chosen method? If it ended after the first favourable morning, the report should disclose that change. Do not describe it as execution of the earlier plan.
The plan may also name an outcome you do not want to worsen: submission errors, enquiry handling time or the proportion of cases needing follow-up questions. Check each against its agreed definition. More forms need not mean the task is being done better if each one then needs manual clarification.
What belongs below the fraction line?
The NIST handbook describes event counts alongside each sample's size when comparing proportions. This basic relationship appears in its chapter on comparing proportions. Before accepting any percentage, identify its numerator, denominator and unit. A count of forms alone does not define a rate.
In our example, 50 divided by 1,000 is 5%, and 60 divided by 1,500 is 4%. This compares events per visit. The difference is minus one percentage point. Relative to 5%, it is a decrease of 20%. Do not label it a "1% decrease": that describes a different change.
Now add that 800 distinct people visited each variant, but people returned more often to B. If the 50 and 60 forms were submitted by 50 and 60 distinct people respectively, the rates are 6.25% and 7.5% of people. B has a higher rate per person even though its rate per visit was lower. There is no mathematical contradiction. We changed the unit and the question.
The plan must determine the appropriate comparison. If a variant was assigned to a person, check how returns were recognised and whether that person retained the variant. If someone could see both A and B, counting unique identifiers in each group does not settle the issue. The report should explain the situation and its implications for the analysis.
Distinguish a count of events from a count of people with an event. One customer may have submitted twice, revising the text after a conversation. Two database records may correctly represent the process, but they do not mean two different customers. Ask for the deduplication definition and an example involving resubmission.
Also record who enters the denominator. "All assigned people", "people who saw the form" and "people who began filling it in" describe different sets. The variant may affect progression between those stages. The report should not give all three calculations the same name of effectiveness rate. Show the set's name and the condition for including a person in each, so a reader can trace the path from assignment to outcome.

The same applies to a survey after a task. Only some participants respond, and a comfort rating can be confused with successful completion. A post-task difficulty and satisfaction survey (in Polish) helps separate these measures. Do not attribute satisfaction to all visitors if it comes from the narrower set of respondents.
Check the time window on both sides of the fraction. A form opened before the trial ended might be submitted later. A later submission may count towards a previously defined group of visits or people if the attribution rule set in advance allows this and the submission falls within an observation window defined in the same way for both variants. Agree how delayed outcomes are attributed and apply the rule consistently, rather than choosing the convenient date for each variant.
Keep the dataset version and preparation method so the report can be reproduced. A model may receive already aggregated numbers without information about returns, errors or exclusions. If the data cannot support the planned measure, the correct answer is that the basis for the calculation is missing. A polished sentence about a trend cannot supply absent records.
Include a column dictionary when passing data to the model. A field named "users" in an export might mean active identifiers on a given day rather than people across the whole trial. Adding daily values does not necessarily remove returns across days. If the measurement documentation does not establish the meaning, ask the data owner. Do not let the model choose a definition from the file header alone.
An uneven split is a question to investigate
Equal numbers of people in the variants are not required in every experiment. The plan may call for a different allocation. Compare expected and observed allocation in the correct unit and using the chosen method. Our 1,000 and 1,500 visits are not automatically proof of unequal allocation of people.
A Microsoft Research publication describes Sample Ratio Mismatch, a difference between expected and observed sample allocation, as a possible signal of data quality problems. Its authors stress the need to identify the cause. That is a limited finding from the paper on diagnosing SRM, not a diagnosis of our example.
During review, ask where assigned units were counted. Before the form was displayed or only after it loaded? If B sometimes failed to open, a dataset containing only successful loads may omit the very cases needed to assess its operation. Do not remove problematic records without documenting the rule and its effect on the comparison.
Treat filtering by device, traffic source and day similarly. A model may show a favourable result in one subset. That is a different conclusion from the outcome of the whole planned test. If further analysis of a group makes sense, mark it as a separate question and examine the basis for the split. Do not retroactively add it to the original hypothesis.
The review should leave a record of exclusions. Instead of "the data were cleaned", state which records were omitted and why. The analyst can assess whether the rule matches the trial design. AI may help prepare the explanation, but its source must be the actual data processing.
From a correct percentage to a statement about the result
Even a correctly calculated difference does not mean a variant "definitely wins". The report should refer to the agreed analysis method and show uncertainty in an appropriate form. If a model received only two totals and invented certainty, ask for its basis. A confident tone cannot replace a specified method and analysis result.
A working report can retain three separate statements: what was observed in the data, what follows from the chosen analysis and what decision the team proposes. That way, correcting a denominator does not require hiding the whole history. The calculation and corresponding conclusion change; the recommendation must be reconsidered against the objective. Version records later show whether the decision relied on the corrected material or the first striking slide.
Separate observation from explanation of cause. "In B, 7.5% of people submitted the form" describes a figure in the fictional dataset. "The guiding questions increased trust" is an interpretation of the mechanism. Even if an experiment supports a particular conclusion about a change in outcome, it does not automatically explain what a user thought.
A separate study can investigate understanding of the screen. A five-second test (in Polish) answers a different question from a randomised behavioural comparison. One result should not substitute for the other. If the report mixes methods, rewrite the conclusions so readers can see which material supports each statement.
Review how the difference is presented too. When AI reads a chart (in Polish), axes, time window and missing points matter. A truncated axis can visually exaggerate a small difference, while a cumulative chart can hide changes in the daily pattern. A correct percentage in the text does not repair a misleading graphic beside it.
Check the decision the report proposes. Rolling out B, continuing the trial and repairing measurement are different decisions. If the definition of a person is missing or data windows differ, a reasonable review outcome may be to complete the material. There is no need to declare a winner just because the final slide has room for a trophy.
If the team starts optimising only the number of enquiries, check whether their quality and handling cost are being neglected. The article on work measures and unwanted behaviour (in Polish) shows why a measure needs to be connected to the purpose of the task. The experiment should support a decision about the form, not make it easier to collect arbitrary records.
A review of AI use and evidence in the report can be arranged with Aeneia Audit. Bring the trial plan, measure definitions and dataset used for the report. In our example, recovering the denominator first corrected the arithmetic and then allowed the right question: are we assessing a visit's effectiveness, or a person's path to submitting an enquiry?








