Independent judgment and AI suggestions: comparing review conditions
Does an AI suggestion help people catch mistakes? Compare an independent first judgment with a suggestion visible from the start, using defined material, order and measures.
Article language: English

In this article
A reviewer who sees a finished AI answer immediately has a different task from someone who first forms an independent judgement and sees the suggestion afterwards. Both workflows involve a person, but give them different conditions for checking the answer.
Do not settle the difference merely by observing that someone once found an error. Nor should you assume an independent first judgement will always be better. Prepare a comparison of working methods using suitably chosen material. The findings should help select a review process, rather than rank employees by attentiveness.
Imagine a fictional team checking descriptions of service scope before sending them to clients. The reference material states whether an offer includes proofreading a language version. AI produces a short decision with a rationale. Some suggestions are correct; others omit a condition or add something outside the scope. The trial uses prepared documents, without sending answers to clients.
What exactly do you want to compare?
In the first condition, the employee receives a document and sees the AI suggestion immediately. They must make a final decision and identify its basis. In the second, they first record their own decision using the same kind of document. Only then do they see the suggestion and have the option to change their decision. This is a change in the order of information, and it needs a clear description.
Write the research question in an ordinary sentence: does making an assessment before seeing the suggestion help this team reject incorrect suggestions in this task, at an acceptable cost in work? Do not replace it with the broad claim “AI weakens thinking”. A small company comparison cannot sustain a conclusion that sweeping.
The study by Lee and colleagues published at CHI 2025 concerned workers' self-reports and the relationship between confidence and reported thinking effort. A survey does not replace an experimental comparison of decision accuracy. It cannot be used to assign a cause to a particular person's mistakes.
Clearly separate the effectiveness of a check from its formal presence. Human approval of AI actions (in Polish) requires the right version, authority and evidence of the decision. All those conditions may be met while the assistance still makes a certain kind of error harder to detect. That is a different subject of assessment from whether the approval button works.
Establish what will stay the same: the documents available for checking, accuracy criteria, instruction length, the way decisions are recorded and the ability to return to the source. If the employee receives an extra explanation of service scope in the second condition, you are no longer comparing order alone. You have added help that the first condition did not receive.
Keep the specific AI suggestions assigned to the documents too. If a new answer is generated during every session, one condition may happen to receive more incorrect suggestions. It will be unclear whether the difference came from the order or from different assistance material. A prepared trial can use suggestions agreed in advance, provided that choice is clearly described. You are then assessing the review process against known examples. Do not present the result as a measurement of the entire product, which generates other answers in everyday work as well.
Provide comparable working conditions in terms of time and interruptions. This does not mean completion time must be identical; it may be precisely what differs. It means not running one condition in a quiet room and the other between phone calls and urgent jobs, then attributing the whole difference to the AI suggestion.
How do you prepare material without letting memory win?
You need answers for which the correct decision can be justified. In the example, one offer includes proofreading only the source file, while another also covers the translated version. The reviewer must identify what the proposal being sent to the client covers and which condition it must retain. The person preparing the reference answer records its basis in a specific passage of the document.
First check the reference answer itself with someone who knows the service. If two competent people interpret the wording differently, clarify the document or exclude the case from accuracy scoring and mark it as ambiguous. Otherwise, the test asks the participant to guess the author's intention.
Do not show the same person an identical example first in one condition and then in the other. They may remember the answer the second time. Prepare comparable sets and distribute their assignments so one condition does not receive only the easier documents. The sets should preserve the important types of difficulty, rather than simply have similar text lengths.
You might prepare two offers requiring the reader to find a condition in a footnote, and two requiring them to connect the scope with the material version. A participant sees different documents but performs a similar kind of task. Keep the document list and condition assignments. Do not move a difficult example into another group only after seeing an unfavourable result.
Vary the order too. Some participants can begin with the immediately visible suggestion, while others begin with independent assessment. Early tasks may teach people how to use the interface; later ones may tire them or reveal the pattern of deliberate errors. Record the order and any prior familiarity with the documents. Using two sets does not remove every difference between people.
In their 2021 paper, Buçinca, Malaya and Gajos compared different conditions of assistance from a simulated AI system. In the Update condition, participants first made their own decision and then saw the suggestion. The study concerned a specific ingredient-substitution task; it does not provide ready-made proof that this order works when reviewing company offers.

Tell participants the purpose of the exercise and that the material may contain errors. Do not reveal the exact location of every trap. Maintain the conditions needed for informed participation, without staging a secret employee test. If the organisation needs a formal study, suitably qualified people should establish its design and rules.
What should you record besides the final “correct”?
In the independent-assessment condition, keep the initial and final decisions. Distinguish improvement after a correct suggestion, deterioration after an incorrect suggestion, maintaining a correct decision and remaining with an error. Counting changes of mind alone does not tell you whether the help was beneficial. People can change their opinion more often and be wrong more often too.
In both conditions, count acceptance of incorrect suggestions and rejection of correct ones. Record whether the employee found the right evidence and how long the task took to finish. A broad “checked” label does not distinguish recognising an error from choosing the right answer by luck. The rationale should be brief but verifiable against the material.
Include an example in which the person's first answer is wrong and the suggestion is right. The review process should allow people to benefit from that help, as well as protect them from mistakes. In the decision record, state what prompted the change: a specific document passage, a condition they noticed or simply the forceful rationale. When a correct suggestion is rejected, you also need to see the cost and reason. A person's own judgement is a point of comparison, but going first does not make it the standard of truth.
Do not confuse the reason for a change with the fact of changing. A participant may say that the suggestion led them back to the right footnote. They may also write that “the AI sounded confident”. These observations help describe what happened, but do not justify diagnosing personality traits. The questions concern task conditions and available information.
Collect ease-of-use ratings separately. The January 2026 preprint by Ghosh and colleagues compares interventions used when reviewing AI plans in writing tasks. It shows a gap between perceived helpfulness and observed reductions in overreliance. The authors identify limits on transferring the findings to other tasks and organisational settings.
This supports separating “do you prefer working this way?” from “do you reach the correct decision?”. A more pleasant experience can be a valuable benefit, and extra effort a real cost. Both belong alongside quality rather than replacing it.
The article on rating AI answers with thumbs (in Polish) develops similar caution in interpreting user reactions. Here, participants and tasks are selected for a trial. In a working tool, only some people may leave reactions, and only on some answers. That is a different way of gathering evidence.
How do you read the result without judging people?
Show the raw counts and scope. If two errors out of eight were missed in one condition and one out of eight in the other, that describes this trial. Do not automatically announce that the new process halves mistakes across the company. The small amount of material and the way it was selected limit the conclusion.
Examine the cases that differ. Does the problem involve locating a condition, understanding the scope or following a suggestion despite finding a contradiction? These mechanisms call for different changes. Better document wording may help in both conditions; delaying the suggestion may affect only the way assistance is used. Do not put everything under “lack of attention”.
Check whether each person performed a similar range of tasks. One may have received more difficult cases or had less experience with the service. In a small trial, show these limits rather than hiding them in an average. Micro, macro and weighted aggregation (in Polish) is a reminder that the method of summarising matters to the decision.
Do not replace independent assessment of the material with another model without checking how it works. If a model helps score rationales, it needs its own criteria and review, as described in evaluating a model used as a judge (in Polish). Otherwise, the scores may reflect the model's preferences rather than the stated review criteria.
Frame conclusions around the organisation of work. You might propose an independent first judgement for cases requiring interpretation of a condition, and a quick AI preview for simple questions about one field. That proposal still needs to be tested on appropriate material. Two successful examples do not make it a universal rule.
What should change after the first trial?
Work with Aeneia Audit can begin with one task the company already requires people to check. Prepare a non-confidential reference document, several different suggestions and a rule for the correct answer. That makes it easier to assess whether you need a different information order, a better source or a clearer requirement itself.
Introduce one justified change and test it on fresh material. Repeating the same traps will increasingly measure the team's memory. Keep observing cost too: an extra step may consume time the employee needs for checking the document. Do not assume that every longer process means greater care.
If the trial reveals missing knowledge about the service, combine the change with your own team practice. A guide to team verification of AI (in Polish) can help. Training should show the basis for a specific decision. Repeating “always check” will not explain how to recognise a missing condition in someone else's document.
Record which task the chosen form of assistance supports, the limits of the trial and when to reassess it.







