Skip to content
Interface language: English
Audit / Signal

AI pretends to be a customer in research: who actually gave the answer?

Distinguish simulated participant suggestions from evidence about real customers in UX research.

Article language: English

Aeneia concept image: a gold-toned, leaf-shaped ornament with iridescent insets rests on dark blue fabric.
In this article

A report says: “Participants want to send their design without creating an account.” It sounds specific. Before commissioning a form redesign, ask who gave those answers. Twelve people who use the printing company, or twelve roles played by a model from a description such as “busy small-business owner”? The same sentence has a different basis in each case.

Here a synthetic participant is an AI-generated character that answers questions or performs a task according to a supplied profile. Such a simulation can help rehearse an interview, find an unclear instruction or suggest another case. Fluent speech alone does not make it an observation of a real customer.

The origin of an answer is easy to lose in a presentation. A working note says “the model suggested”; a slide says “research indicates”; after the meeting it becomes “customers confirmed”. The sentence has acquired witnesses who never attended an interview. Review should begin with how the material was produced, before the team counts its topics.

What was the simulation meant to check?

Consider a hypothetical printing company designing a new way to request file corrections. The team wants to know whether a customer understands the difference between sending a comment and approving the design for print. These are separate tasks. A simulation may suggest possible mistakes for a scenario, but it does not show how many customers actually make them.

Before using a tool, record what the answers will be used for. “Prepare interview questions” requires a different review from “choose the better interface”. In the first case, review the questions for clarity and relevance to the decision. Their reception by participants must be checked later. The second requires evidence of the behaviour of appropriately selected people. Changing the goal after seeing convincing answers should prompt a new assessment of the material.

A model can be asked how the label “Approve” might be interpreted. Treat the result as a list of possible readings. A designer checks them in the screen's context and then selects a trial with people. “Possible confusion between approving a comment and approving print” is a useful hypothesis. “Most customers confuse these actions” needs a different basis.

A simulated conversation can also help a moderator rehearse. If the model gives a general answer, the moderator can prepare a neutral question about the most recent real example. They are evaluating their own scenario and follow-up technique. They should not count the simulated answer as data about the printer's customers. Rehearsing with a colleague does not automatically make that colleague a representative of the segment either.

A generated participant can receive constraints: no printing vocabulary, no finished file or no remembered order number. Check whether the tool maintains them. If it supplies the missing number on its own a moment later, the simulation does not cover the intended problem. That is a finding about the trial, not about customer ability.

If a simulation performs a task in a browser, record what information it can access. A model may receive text hidden from an ordinary user, full element names or an instruction identifying the correct button. “Completed the task” would then not show that the visible screen clearly guides a person. A useful technical check of the form remains a different trial from research into comprehension. An automation's movement log does not settle how a customer reads the screen.

The scope should also be clear to the person commissioning the work. “AI-assisted scenario review” describes this material more accurately than “12 interviews”. If real interviews will happen later, show them as the next stage with separate participant selection. The rules for recruiting appropriate UX research participants (in Polish) still apply.

A similar answer does not mean the same evidence

A design study on augmented reality for education compared synthetic material with teachers' answers. The authors described supporting usefulness in some open responses and discrepancies on more structured questions. This is a limited result from one design study, not confirmation that any persona accurately replaces your company's customers.

Another publication compares simulated and real users speaking with an agent about order-service tasks. Simulation results did not reliably reproduce those obtained with people. The authors of the Lost in Simulation preprint examine a particular task set and configuration; they did not test our hypothetical printer's form.

The practical review conclusion is narrower than “AI is unsuitable for research”. If someone wants to use a simulation instead of people for a specific decision, they should show an appropriate comparison and the limits of that use. Matching topics in one task does not automatically confirm an answer distribution in another.

At the printer, a model may mention speed, price and convenience because they are plausible topics about a service. A real customer may instead describe a file prepared by an absent designer or the need for a business partner's approval. We do not know in advance which detail will matter. That is why we look beyond the description of a created role.

Write a hypothesis so that it can be challenged. Instead of “customers need convenience”, ask whether the person requesting a correction knows who may approve printing. In a real trial, ask them to send a comment and then identify the order status. Observing that a customer distinguishes the two actions and correctly describes the state weakens the suspected confusion. Later statements need not all be fitted to the problem suggested by the model.

These limits are visible when observing website use at work (in Polish). A person may interrupt a task, answer a call, consult a document or ask a colleague for confirmation. A model may describe similar events if it invents them or receives them in a prompt. That is different material from seeing them occur during research.

Aeneia conceptual photography

Also distinguish a statement from an action. “I would certainly click the big button” is not a record of a completed task even when a person says it. A generated statement adds another layer of uncertainty: it comes from simulation. A moderated prototype test (in Polish) checks the progress of a specific task and understanding of the interface rather than stopping at an attractive answer.

A profile may contain the answer already

Read the instruction that created the persona. If it says “the business owner dislikes creating accounts and values fast file submission”, a later complaint about registration is not an independent discovery. The condition was written into the character. It belongs among assumptions in the report, not new market findings.

A task description with an embedded judgement works similarly. “What is wrong with the excessively long form?” already assumes the form is too long and causes a problem. Replace it with a question about the task and treat length as a feature to investigate. Neutral questions matter with people too; simulation does not remove that obligation.

Record what material the model received. If it saw earlier interviews, it may reproduce their topics. That can help rehearse scenarios, but agreement with the same material is insufficient evidence that it can predict new research. Testing a new use requires material the model did not receive while preparing its answer.

Choose profile details for the question. For a design sent to print, approval role and file access may matter. A decorative biography about favourite coffee adds character but does not confirm process knowledge. More detail does not automatically mean more credibility.

Do not infer behaviour from a group label without appropriate data. “Foreign customer” includes different languages, roles and experience with printing. If a decision concerns a local ordering practice, contact the appropriate people and sources. The article on B2B customer interviews in Asian markets (in Polish) gives an example of selecting such conversations. A detailed persona does not replace understanding real conditions.

Check sensitivity to a small prompt change. If replacing “cautious” with “experienced” completely changes a recommendation, record that dependence. The trial may show that the material depends heavily on how a character is described. It does not establish which version better represents customers; that requires additional evidence.

Repeated generation can help explore possible suggestions. It does not turn repeated runs into separately recruited people. Report the number of generations as generations. Twelve names and twelve portraits do not by themselves increase data independence.

During analysis, do not count several versions of the same sentence as separate needs. “I want to send the file faster”, “submission should be efficient” and “the form should work without delay” may come from one instruction and describe one topic. Combine them into a working hypothesis while retaining the generated examples. A long list of paraphrases should not determine feature priority merely because the model elaborated on that topic most.

How to report a result without creating imaginary participants

In working materials, clearly label each statement's origin. Distinguish an actual interview record, a synthetic answer and a researcher's comment. Do not put them into one unlabeled quotation list even if you intend to organise it later. Such lists travel easily into presentations.

For a simulation finding, show the assumption, suggestion and question to investigate. For the printer: the profile assumes no access to the final file; the model suggests a draft save; interviews should establish when a real customer receives the file and who approves it. This helps plan research without pretending to be its result.

Until interviews occur, assign each hypothesis an owner and a place in the scenario. After research, add what was actually observed, which observations contradicted the assumption and what remains to check. Do not retroactively revise the original AI suggestion to make it appear to be a correct forecast. The preserved record distinguishes an idea useful for preparation from one unsuited to the printer's work. Both outcomes help the team use the tool more sensibly next time.

A synthetic quotation can illustrate a scenario if labelled as such. Do not attach a name and title implying a real interview. The report's reader should recognise the source even if a slide is separated from the presentation. A short label beside the quotation is more reliable than an explanation hidden only in the introduction.

If a passage's origin has been lost, find a record of how it was produced. An AI text detector (in Polish) cannot replace source history. Judging writing style does not establish whether a particular customer was interviewed that day. Find the recording, note or generation log, and state honestly when it cannot be found.

Before a product decision, record what is established and what remains a hypothesis. A team can immediately correct an obvious wording ambiguity it has identified. A choice about login, approval sequence or a new service scope needs evidence appropriate to that decision. Explain the reason for a change rather than crediting nonexistent participants.

For example, the team may select “Send comments on the design” because it describes the button's effect more precisely than “Approve”. That design rationale can be assessed without a story about twelve customers. Whether the new label and surrounding screen actually prevent confusion during work remains a separate question to test. Separating the decisions lets the team improve the product without overstating the available evidence.

Such a review can be discussed as part of Aeneia Audit's assessment of AI use. The conversation starts with the tool, material supplied to it, result and decision based on that result. Simulation can remain a useful assistant in the team's workshop. For conclusions about customers, we still check who gave the answers.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear