AI reads survey answers: the loudest theme is not always the most common
How to check AI analysis of open-ended answers: counting units, multiple themes, quotations and the limits of conclusions.
Article language: English

In this article
"The service was excellent, but my parcel seems to have toured half the country." One answer, at least two themes, and a slightly weary respondent. A model might label it positive because it noticed the praise, or negative because it focused on delivery. Either result leaves the team wondering what to improve. Meanwhile, the parcel may have had a richer travel itinerary than its owner.
AI can help organise open-ended survey responses: suggest themes, find passages and draft a summary. An audit should follow the analysis from each response to the final conclusion. Along that path, qualifications can disappear, counts can change and the most colourful sentence can gain undue weight.
You do not have to assess the whole study at once. Start with one decision the team wants to make from the answers. To improve delivery, you need the type of problem, its context and the unit being counted. A general "negative response" label is too broad to select a concrete action.
An answer, a person and a mention are different units
Imagine a fictional post-purchase survey. One person mentions delivery three times while describing successive delivery attempts. Another uses one sentence. If you count mentions, the first response may weigh more. If you count people reporting a problem, each contributes one observation of that kind. Neither calculation should be a hidden model setting.
Before grouping, record what the result should mean. "Ten responses mention delivery" differs from "ten delivery problems" and from "ten customers had a problem". One response may describe several events, and one person may submit two responses. If you cannot establish that connection, do not add it to the report.
Keep response identifiers and references to the passages supporting each theme assignment. In the working analysis, a reviewer should be able to move from a group back to individual responses. A summary without that route can look persuasive yet be difficult to check when someone asks what supports a conclusion.
AI-assisted record matching (in Polish) needs a separate check if you first remove supposedly duplicate responses. Similar wording does not always mean the same person. Two customers may describe the same problem in identical words. Removing one voice changes the result, even when the final sentence is perfectly written.
Count a small example by hand. Suppose five people each submitted one response. Two mention delivery, and one of those two returns to it three times. "Two out of five people" answers a question about participants. A count based on passages answers a different question and needs a different denominator. Do not compare those values as if they expressed the same frequency. If one response belongs to several themes, explain that group totals can exceed the number of responses. That need not be an error; readers need to know about multiple assignments to interpret the table.
Then divide one long response into shorter passages without changing its content. If the number of people reporting the problem rises, the counting mechanism is confusing passages with participants. This test identifies a concrete cause of the difference instead of debating whether the summary sounds credible enough.
Also establish the denominator. Are you counting all responses, only those to a particular question, or only responses with a recognised theme? If the table comes from a database, check what AI-generated SQL actually counts (in Polish): rows after joining tables need not equal responses or people. Show blank fields separately. Otherwise, a difference between two surveys may reflect a change in how blanks were excluded rather than customers' experience.
Look at what disappears when a label is assigned
Microsoft's sentiment analysis documentation distinguishes document or sentence sentiment from opinions about particular aspects. This is a useful conceptual boundary. The tone of a whole response need not match the opinion about every part of a service.
In our survey, service may receive praise while delivery is criticised. If the model assigns only one label, check how it is supposed to retain both themes. You can allow multiple themes or split the response into passages while keeping a shared identifier. The choice depends on the aim, but it should be explicit and tested.
Prepare a small control set with simple and complex responses. Include negation, a comparison with an earlier purchase, conditional praise and a joke. "The courier found us this time" can mean something other than general praise for delivery. Before seeing the model's output, a reviewer should mark the expected themes and limits of interpretation.
Give part of the set to a second person without showing the model's labels. Compare the themes identified and the points of disagreement. If the reviewers differ in much the same way as the model and the first reviewer, the category definition may need work. Do not put every such disagreement in one bucket of tool errors; first establish whether the material supports a decision.
Record "unknown" when the response does not support a confident assessment. Irony is not always unambiguous. The point of checking is not to prove that humans always understand text flawlessly. It is to show uncertainty where an automatic label could turn it into a seemingly precise number.
Examine responses the model omitted. They may contain typos, mix languages or describe a problem without its usual name. "I had to stay home for a second day" may concern delivery even though it does not mention a courier. Rejected or unassigned responses are part of the result, not a technical bin outside the report.

When testing AI message classification (in Polish), we follow one case. Survey analysis adds a collective effect: a series of similar mistakes can change the apparent picture of the whole service. Alongside individual labels, compare group counts after manual correction.
A theme name can suggest a cause
"Incompetent delivery providers" already contains a judgement. "Comments on timing and delivery" describes the scope of responses more cautiously. Reports of delay do not automatically establish its cause. An audit should check whether group names stay close to the material or take the place of an operational diagnosis.
GOV.UK's guide to analysing a research session separates observations of what was seen or heard from interpretation and groups material into themes. Keep similar clarity in a survey: the participant's response, the assigned theme and the team's conclusion should be distinguishable.
For one group, show several different responses and a borderline case. If only the most striking quote fits, the name may be too narrow or the group too broad. Also check whether the model creates separate categories for synonyms. "Hard to collect the parcel" and "inconvenient collection hours" may need joint analysis, though they do not always describe the same event.
Ask about the absence of a theme in a particular response. Silence about delivery does not mean satisfaction with delivery, especially when the survey asked generally about the purchase. Distinguish explicit praise, explicit criticism and no statement about an aspect. Otherwise the model may inflate the satisfied group by attributing an opinion people did not express. Check this distinction on several responses concerned only with another part of the service.
Changing category names halfway through requires revisiting earlier responses. Otherwise the second part of the survey may be assigned under more precise rules than the first. Keep the theme-set version and record when the material was reassessed. That permits comparison without pretending the criteria were identical throughout.
Do not force a rare case into a large group merely to reduce the number of colours on a chart. A single account of being unable to use a service may deserve separate investigation even if it does not change the ranking of themes. Frequency helps establish prevalence in the collected responses; it does not by itself decide the problem's importance to the business.
A summary must also be traceable backwards
Select a few sentences from the final summary and find their basis. Does "customers do not understand collection rules" come from statements about the rules, or from a few descriptions of delay? Has the model added a cause participants did not give? Each difference should enter the assessment rather than disappear into a general note saying verification is needed.
Compare short and long responses too. A model may produce a richer summary where it received more words, even though short responses describe the same problem. Check whether the final report preserves the number of responses in the group or promotes the longest stories. Comment length should not quietly replace the agreed counting unit.
Read a group after removing its most vivid quote. If its description changes completely, check whether the earlier title was tailored to that one sentence. This tests the robustness of an interpretation; it is not a way to remove inconvenient voices. The original material remains part of the analysis.
Check that quotations are exact and attributed to the right responses. An abbreviation can change meaning, especially if it removes "but", a negation or a condition. Distinguish a verbatim passage from a paraphrase in the report. Do not let an elegantly edited model response pass as the survey participant's words.
Use only the data appropriate to the purpose. Checking a label does not always require a customer's complete history. In logging an AI process without excess data (in Polish), an identifier and controlled access to the material can help reconstruct the process without multiplying copies of responses across every working table.
For a review with Aeneia Audit, prepare a sample of responses, the label set and the finished summary. This allows all three layers to be compared instead of discussing only the style of the final report.
What the team can say after the review
If the survey does not support a conclusion about the whole customer population, automatically organising the material does not expand the scope of inference. In the summary, use the appropriate scope: "in the collected responses", with the counting unit stated. Do not infer representativeness from the number of paragraphs, a confident tone or an automatically generated chart.
In an AI pilot (in Polish), assess the cost of corrections too. A tool may quickly create draft groups yet require lengthy quote checking. Record where it saves work and where it shifts work. This helps establish a sensible role for the solution in the team.
Survey conclusions may later influence product recommendations in a catalogue (in Polish). Before a rare, striking comment changes the order of an entire offer, check the basis for that decision against other process data. A single response may accurately identify a problem without describing every customer's preferences.
A well-reviewed analysis leaves the team with themes whose meaning can be explained, numbers that can be reconstructed and questions for further checking. The model then helps read a larger collection rather than replacing it with a polished story. Even the wittiest comment about a travelling parcel remains one person's voice.








