Skip to content
Interface language: English
Audit / Signal

AI answer ratings: who clicked, and who is missing from the report

What do thumbs on AI answers actually measure? Define the denominator, link each reaction to its result and review unrated answers before describing the whole service.

Article language: English

A hand beside a gold-coloured mechanism with a suspended clear drop-shaped object and a pale sphere.
In this article

The assistant's report shows eighty per cent positive ratings. Before drawing a conclusion, check whether that percentage refers to answers, users, completed cases or clicks.

A user's rating is useful information about their experience. It does not automatically become an assessment of the correctness of every answer. Particular people leave voluntary reactions on particular results, at the moment when they see a way to rate them. Everyone else may be happy, disappointed, busy or unable to respond. The absence of a thumb does not tell those explanations apart.

Let us follow a fictional assistant that helps staff find company sales materials. In a selected period, it displayed a thousand answers. A hundred had a current reaction: eighty positive and twenty negative. For this calculation, we count at most one current rating for each answer. We are not evaluating a real tool. These simple numbers help check what a report can legitimately say and how to plan a further review.

What does the eighty per cent measure?

We can say that positive reactions accounted for eighty out of a hundred reactions. We can also report the share of answers rated: a hundred out of a thousand, or ten per cent. The eighty positively rated answers make up eight per cent of the full thousand. Each number has a different denominator and answers a different question.

It does not follow that the other nine hundred and twenty answers were bad. Nine hundred have no reaction, while twenty received a negative one. Those states need to stay separate. If software turns an empty rating into zero, it creates an apparent result that users never supplied.

We do not know the proportion of satisfied people either. One employee may have rated thirty answers, while everyone else rated one each. A hundred reactions would then not be a sample of a hundred independent users. A report about people needs an appropriate way to distinguish them, within the agreed data scope. Do not collect additional information unnecessarily for this purpose.

If distinguishing people is possible within the permitted scope, show how concentrated the reactions are too. A report based mainly on a few regular users may describe their experience well while saying little about people trying the assistant for the first time. Do not try to fix that by giving every thumb an arbitrary weight. First describe the groups' actual participation and the question you are answering.

AAPOR's discussion of response rates and survey quality warns against treating the response rate alone as a decisive measure of quality. It calls for other evidence and a transparent method. This is a reference about surveys; it does not turn voluntary thumbs on an assistant into a representative poll.

Name the population covered by the report: all assistant answers displayed in the selected period, through the specified channel and version. Check that the count excludes internal tests, the author's previews and the same answer refreshed several times. Define the event before calculating percentages.

You also need to establish whether every user could rate an answer. Buttons may be available in the panel but absent from a message forwarded to a messaging app. A missing reaction in the second channel then has a technical explanation. A combined report should disclose that rather than compare the supposed willingness of two groups to leave ratings.

Do not make a thumb compulsory simply to increase the rating rate. A forced reaction may express a wish to close the window. First decide what information is needed and at what point a person can meaningfully provide it. More records do not guarantee more knowledge about the service's usefulness.

Which answer does the reaction belong to?

In a controlled example, generate two answers, then rate the first while the second is already visible. Check the recorded data. The rating should point to the result beside which the person pressed the button. A conversation ID alone is not enough if that conversation contains a dozen different suggestions.

Microsoft Foundry's documentation on logging feedback describes associating a rating with a specific trace or span of execution. The feature is marked as preview. We refer to the principle of correlating data, without recommending this version for production or claiming that any assistant supports it.

Keep the answer version with the rating, along with what the button refers to. A user may press a thumb after editing the text or choosing another document from the search results. You then need to understand whether they are rating the original suggestion, the result after their own work or the whole conversation. Calling a field “feedback” does not explain that for the team.

Test a changed reaction too. A person initially chooses positive, then opens the file, discovers that it is an old offer and changes the rating to negative. Agree whether the report shows an answer's current rating or the history of events. Two clicks by the same person must not quietly masquerade as two ratings of different results.

Distinguish removing a rating from choosing its opposite if the interface allows a selection to be cancelled. No current reaction and a negative reaction convey different information. Check the flow in the export, not just on screen. An icon may return to a neutral colour while the system still counts the earlier event.

Include returning after a break and rating an older answer in the tests. If one report assigns a reaction to the day of the click and another to the day the answer was generated, their period comparisons cover different things. Both perspectives can be useful, but they need names and must remain distinguishable.

Aeneia conceptual photography

Does a positive reaction confirm that the task was completed?

In our example, a user may appreciate a quick, polite answer that leads to an outdated file. They may also rate the correct information that the requested material does not yet exist negatively. Agreement with the source, perceived usefulness and satisfaction are different dimensions. You do not need to discard the reactions to check the result independently.

In its May 2025 account of excessive agreeableness, OpenAI said that user-preference signals may have favoured answers more agreeable to users, while positive tests did not catch the full behavioural problem. This is a provider's account of its own incident, not a diagnosis of your assistant or evidence that every thumb is harmful.

Prepare a separate check of the task outcome. For finding an offer, check whether the file exists, is the right version and answers the scope of the question. Record the basis for that judgment. Comparing the outcome with the reaction will then reveal cases of satisfaction despite an error and dissatisfaction with a correct answer.

Do not ask users about something they have not yet had a chance to check. A thumb selected before opening the material describes an earlier moment in the experience. If you want to know whether the file helped them prepare an offer, you need information after that action. The article on post-task surveys, difficulty and satisfaction (in Polish) explains how to design that question.

When adding a comment field, make it easy to describe the problem without suggesting one cause. “What influenced your rating?” leaves more room than a compulsory “the model made a mistake” option. The cause might have been an old file being retrieved, missing access or a long wait. Do not require the user to work out the solution's internal architecture.

Before using reactions to change an instruction, return to the provenance of AI feedback data. A positive thumb does not turn the entire text into verified reference material. Confirmation that it was useful for one purpose does not validate every claim it contains. The related topic is explained in the article on AI outputs feeding back into inputs (in Polish).

How can you check what nobody rated?

Select unrated answers for independent review too. Do not limit the review to the twenty negative examples: that would leave unrated answers out of the review. Describe how the sample was selected and its scope: period, channel, task type and tool version. Keep it linked to the answer being assessed.

You can spread the review across frequent and less common kinds of question so that no important group disappears. If the selection deliberately changes the proportions, show the groups' results separately. Do not present the average from a deliberately balanced sample as representative of all traffic without accounting for how the sample was selected and the groups' proportions. The article on aggregating AI evaluation results across task groups (in Polish) helps trace that boundary.

In the review, distinguish a missing reaction from a result that was never delivered. The user may have abandoned the task before the assistant showed anything. That case is outside the thousand displayed answers in our example, but it may matter when assessing the service as a whole. The report should disclose both scopes if the company wants to use them for a joint decision.

Do not invent a reason for silence. Finding an error in an unrated answer does not prove that this was why the person did not press the button. You can check the quality of the material and ask separately about the reasons for reacting or not reacting in an agreed study. This protects the report from an appealing story that the data do not support.

What reviewers know before they begin matters too. Someone told “this answer received a negative thumb” starts with a different cue from someone given material without its rating. If the aim is an independent check of correctness, consider hiding the reaction until the reviewer reaches their own judgment. The article on independent human assessment before an AI suggestion (in Polish) discusses the conditions for such a check.

Keep the types of error found and the evidence for them. Another percentage labelled “auditor agreement” is not enough. You need to know whether a fix concerns source versions, access, the way answers are produced or the rating question itself. Each problem leads to different work.

After a fix, return to the identified examples and check their new counterparts in current use. An increase in positive reactions alone does not confirm that the old file has been removed from search. You need to read the correct document and check whether the answer uses its current version. Continue observing user ratings too. That lets you see the change in correctness separately from whether people found the new route useful.

What should the decision-maker see?

The report should include the number of answers displayed, the number rated, the reaction breakdown and the scope of independent checks. For our fictional thousand, do not turn a ten per cent reaction rate into a guess about the other ninety per cent. State which information was observed and which has not yet been collected.

When comparing periods, check for interface changes. A more visible button can increase the number of ratings without changing answer quality. A new request for a comment can reduce the proportion of people who respond. Record the version of the rating collection method alongside the assistant's version. Do not attribute every movement in the metric to the model.

Compare the rating with the time and work needed to use the result. Measuring time spent working with AI includes subsequent corrections. Satisfaction with a quick suggestion can coexist with a long search for the right document. Improvement then needs to address the whole path, not just the moment when the text appears. Further guidance is available in the article on measuring the time spent working with AI (in Polish).

At Aeneia Audit, this review can start with one report and a few non-confidential examples covering different rating states. It helps if the team can reconstruct where the count comes from and what each click refers to.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear