Skip to content
Interface language: English
Design / Signal

Post-task surveys: what does ‘easy’ really tell you?

Design a short survey that distinguishes perceived difficulty, satisfaction and task success. Choose clear questions, response options and timing.

Article language: English

A person lifts a sheet from an open sample case on a wooden table beside a simple meal.
In this article

A participant selected “very easy” after choosing a photographic backdrop 160 centimetres wide. The task was to find one that would fit a space 140 centimetres wide. The survey looks reassuring; the researcher has more to investigate. The two observations can both be true: the rating describes how the task felt, while the selected product can be checked against the task requirement.

This is a fictional example from Mira, an equipment rental business trying to improve product selection on its website. A brief question after a task can help explain the experience, provided the team knows what it is asking. Perceived difficulty, satisfaction and actual success are different outcomes. Keeping them separate makes the findings more useful for design decisions.

Before writing a survey, finish this sentence: “The answer will help us decide...” If the team cannot say what it will do with the answer, another question may only make participation longer. A questionnaire about the whole brand will tell you little about whether someone understood the width of the backdrop. Collect one specific response and connect it to what the person actually did.

“Easy and enjoyable” asks two questions

“How easy and enjoyable was it to choose a product?” combines different experiences. The choice might have been easy but tedious. The page might have looked attractive while requiring several trips back. The participant has to choose one number for two things, and the researcher cannot tell which one shaped the answer or how the person weighed them.

AAPOR's survey research guidance recommends asking about one concept at a time, using simple language and choosing response options carefully. It does not provide a validated questionnaire for every website. At Mira, the first step is to define what the team's own question needs to reveal, then check how people understand it.

One possible post-task question is: “How easy or difficult was it to find a backdrop that met the requirements?” It refers to the task just completed. If the decision concerns whether people can read the width unaided, the team could ask specifically about that step. Do not narrow the question's scope only after seeing the answers: a favourable overall rating is easy to misattribute to one detail.

If satisfaction with the process matters to the decision, ask about it separately and use that name in the report. Do not label a result “website effectiveness” simply because people rated the experience highly. A question about intention to return also captures a stated intention, not a confirmed future visit.

The archival Common Industry Format for Usability Test Reports, version 2.0 from 2001, distinguishes effectiveness in reaching a goal, the resources used and subjective satisfaction. We use it here to clarify those concepts, not as a ready-made survey. In Mira's example, the width of the selected product is checked independently of how the participant felt.

During a pilot, ask a person what they think they are being asked to rate before they choose an answer. If they discuss the quality of the backdrop when the question was meant to concern finding it, you have found a difference in meaning. Do not solve it by coaching every participant in the main study. Revise the wording and check it with another person. This is a test of the question, so keep its answers separate from the main study results.

Define task success before the session. For Mira, the backdrop must fit the stated space and meet the other conditions in the task. Record the actual selection and the basis for treating it as correct. If the conditions do not let you determine whether a product fits, the test itself also needs work. A survey cannot supply a missing success criterion.

The scale is part of the question

For your own difficulty question, you might offer five labelled answers: “very difficult”, “somewhat difficult”, “neither easy nor difficult”, “somewhat easy” and “very easy”. This is a working example of a scale, not a standardised instrument. In the pilot, check whether people understand the middle and both ends, and whether the options read naturally with the question.

Do not leave numbers unexplained. One person may take one to mean the best rating; another may take it to mean the worst. Record the direction of the scale and retain it in the export and report. If a chart reverses the order, its labels still need to make the meaning clear. A mathematically correct average calculated from reversed codes describes something other than the intended experience.

Distinguish a middle response from having no basis for an assessment. “Neither easy nor difficult” does not mean “I did not complete the task” or “I prefer not to answer”. Decide which options outside the scale are needed and how they will be recorded. Do not treat a missing answer as zero. Otherwise, a problem with the survey form can look like an especially low rating of the website.

Fit the control to the device. On a phone, every label should be readable and clearly associated with its option. If a wide scale must be scrolled, a person may miss part of the range. Provide an unambiguous way to choose and test the question's accessibility rather than assuming face icons will convey the precise meaning of every point.

When using an existing documented measurement instrument, establish its intended scope and language version. Changing the wording, number of responses or scoring may call for a different interpretation. Do not give an improvised mixture the name of a recognised questionnaire. Show exactly what participants saw and how the reported result was calculated.

Aeneia conceptual photography

Test how data is stored before launch. Are selecting the first option and skipping the question recorded as different values? If someone returns to the screen, is the answer updated or appended as a separate record? Does the export retain its connection to the task? You can run these checks with test data before collecting actual responses.

Ask about the moment you want to assess

A question immediately after choosing the backdrop concerns the fresh experience of that task. A question after the whole session may also reflect delivery, contact, the catalogue's appearance or fatigue from earlier exercises. Choose the moment to match the goal. When asking about one task, name it so the participant does not have to reconstruct which of several choices you mean.

Record any help given as well. A person whom the researcher directed to the correct width may honestly rate the remaining choice as easy. That rating does not describe an unaided journey through the entire task. Keep the timing and substance of the prompt with the result. If the goal was to find the product independently, mark an assisted outcome separately; the survey response remains the participant's own assessment. There is no need to remove it or lower the rating.

Consider when you reveal the outcome. If you first tell a participant that the chosen backdrop is too wide, a later rating is given with that knowledge. That may be a valid question, but it is different from an immediate assessment after the task. Record the order in the procedure. Do not combine responses from the two moments into one series without disclosing the difference.

In an unmoderated study (in Polish), the instruction and scale must make sense without a facilitator's explanation. In a live session, the researcher should not paraphrase the question differently for each person either. If the question repeatedly needs explanation, revise and retest it. Record any extra explanation so that the conditions for each participant remain clear during analysis.

You can add a short open question: “What most influenced that rating?” It does not assume that a problem occurred. The response can reveal whether the person was thinking about labels, time, appearance or their own familiarity with the products. It is still a participant's account, not a complete explanation of every observed action. Keep it with the task record and product choice.

To assess understanding of a specification, use a separate task, such as asking someone to explain the width in their own words. That is the role of a content comprehension test (in Polish). An ease rating does not contain the correct answer about the product. A question that checks a skill in training material (in Polish) also serves a different purpose from a survey about the experience.

A satisfied participant may still need better information

In the report, place the outcome and the stated rating from the same session side by side. A correct result rated difficult suggests a different line of enquiry from a mistake rated easy. In the first case, examine effort and workarounds; in the second, examine how the page communicates the conditions for a choice. Do not change the participant's rating to make it fit an expected success story.

At Mira, a person might have called the choice easy because they quickly found an attractive backdrop and missed its width. That is a hypothesis to check against the session and conversation. It does not follow automatically from a high rating. Connecting the answer to the chosen variant lets the team return to the question of information hierarchy on the product card.

When one person performs several tasks, retain that connection in the data. Five answers from one participant are not five independent people. Record task order: an earlier exercise may have taught the person how to read dimensions. If ratings change over successive tasks, consider practice and fatigue alongside differences between screens. The ratings alone will not establish which factor was responsible.

Do not hide the sample size behind an average. Show the distribution of responses, the number of ratings and missing values. In a small sample, one extreme response can shift the summary considerably. Two ratings coded two and four have an average of three, as do two ratings of three. In the first set, people chose different sides of the scale; in the second, both chose its middle. This is an illustrative calculation, not a finding from a Mira study. The distribution reveals that difference before a team reduces both experiences to the same bar on a chart. Record whom you invited and who answered. A survey available only after successful completion does not describe the experience of people who became stuck or left earlier.

Using AI to analyse open survey responses (in Polish) may help organise comments, but it should not invent a reason for a missing rating. Keep the original text and its connection to the task. If a model summarises all high scores as “the website is effective”, return to the meaning of the question and the choices people actually made.

For each language version, check the whole question together with its response labels. A literal translation of the middle option may not carry the intended meaning. Record the language and form version with the data, and pilot comprehension with people who use that version. Do not present your own scale as a validated translation of a recognised instrument. Otherwise, the report promises more than the work can support.

When comparing variants or later releases, keep the question, scale and timing consistent, or explicitly describe what changed. A higher score after changing the question is not, on its own, evidence that the design improved. An A/B test conclusion (in Polish) requires checking what was compared and how the result was calculated. General satisfaction is not a substitute for a measure of the goal.

Connect the survey plan to an experiment hypothesis card (in Polish): what change should help, with which task, and what would indicate an effect? You can discuss your website project and its possible scope with Aeneia Design. At Mira, a sound decision might mean presenting the width more clearly while keeping the selection process convenient. A high score is still useful, provided the team knows which question it answers.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear