Skip to content
Interface language: English
Audit / Signal

AI writes training questions: a correct answer key can test the wrong thing

Review AI-generated training questions against the skill, instruction and reasons behind the answer key.

Article language: English

A baker divides warm bread in a kitchen at night, with navy cupboards and a stone hearth.
In this article

In a fictional example of a generated quiz, the answer key says: “Choose the current catalogue.” The trouble is that the question asked: “What should you choose before sending the current catalogue?” The participant did not have to identify a version, check a date or open the library. Spotting the repeated word “current” was enough. The question generator received an assignment; the learner received a small contest in finding repetitions.

At the fictional company Karda, a team is learning to use a new materials library. AI is to help prepare short exercises. The aim is to check whether an employee can select the catalogue matching the recipient's language and the campaign date. This is ordinary tool training, not a professional examination or an employment decision. Review the questions as training material with a purpose, a source and an explanation for each answer.

The answer key starts before the question is written

First describe the action the participant should learn. “Knows the catalogue” is too broad. “Uses campaign information to select the agreed language and the version valid on the campaign start date” makes a concrete case possible. State what the learner may use during the exercise. If the library and instructions are open during work, the task need not test memorisation of names.

In Karda's exercises, distinguish reading information from applying a rule. “What is the date field called?” tests a different action from choosing among catalogues with different dates. Both may be useful, but the first result is not evidence of the second skill. Training a team to verify AI (in Polish) also considers matching an exercise to a role.

Prepare an approved extract from the instruction and identify its version. In this example the rule is: choose the language agreed with the recipient and the edition valid on the campaign's start date. A file's upload date is not its effective date. This is an author's assumption for a fictional process. In a real company, do not invent such a rule merely because it makes a quiz easy to write.

Now build a case. The campaign starts on 15 June and the recipient has confirmed Polish. The library holds three catalogues from the same year: Polish valid from 1 May, Polish valid from 1 July and English valid from 1 May. For this fictional case, assume the Polish May catalogue was uploaded on 1 June, the Polish July catalogue on 10 June and the English May catalogue on 1 May. Under the stated rule, the Polish May catalogue is correct. The July edition is not yet valid and English is not the agreed language.

The question should state the campaign date rather than ask generally for the “current” version. Without a date, a participant may interpret that word as the day of the exercise. Here the relevant date is 15 June regardless of when training occurs. Also say whether to choose one answer or all that apply. Changing this instruction changes how selections are interpreted and may require a different key.

Keep the reason for choosing the May catalogue with the question. The key “A” alone is not enough for a later review, especially if the platform changes the order of options. Link the answer to the option's content and the relevant passage in the rule. Preserve the case conditions too: removing the campaign date can make the question impossible to resolve.

The 2024 QGEval study separates the linguistic quality of generated questions from their consistency with the material, answerability and fit with a supplied answer. It concerns a particular set of questions and models. We use the distinction to design a review, without transferring its results to Karda's present tool.

For this exercise that means two separate readings. One checks whether the question is understandable. The other checks whether the supplied instruction supports a justified choice of catalogue. A fluent sentence can pass the first review and fail the second. Do not average the results so that good grammar cancels an incorrect key.

Record the limits of what is known. If the instruction does not explain what to do when the required language is missing, a question about that case should wait for the process owner's decision. The model cannot decide on its own that an English version may be sent. Sometimes the correct review outcome is to find a gap in the instruction rather than repair one word in the quiz.

Answers should reveal a misconception, not give away the key

The University of Waterloo's Centre for Teaching Excellence recommends, among other things, a clear problem, one best answer and plausible wrong options. These are guidelines for designing multiple-choice questions. Karda applies them to recognition of a specific rule, not formal certification of competence.

Wrong options that are too easy allow learners to bypass the necessary knowledge. If the correct catalogue sits beside “send an empty file” and “delete the entire library”, a participant can reject those without checking dates. Better options reflect actual mistakes: choosing the newest file, the right date in the wrong language, or the right language with a future effective date.

That does not mean setting traps. The participant should resolve the problem using the rule they learned. Avoid several negatives in one sentence unless that is the skill being taught. If decoding the question is laborious, you may start testing patience with text rather than work with the catalogue. That is a separate skill nobody asked for here.

Review each option and record why it does or does not fit the conditions. “Choose the newest version” and “choose the version in force” may describe the same file in a simple case. To test the difference, create a case in which the two rules actually lead to different choices. Different wording alone does not make two actions mutually exclusive.

Aeneia conceptual photography

Check whether form reveals the key. The correct answer should not be the only long, carefully written sentence among three short jokes. Read the question with each option separately. Notice grammar, file labels and words repeated only in the correct option. These clues can turn the exercise into guessing the author's intention.

A review card for the catalogue question can contain four short fields:

  • Skill: identify language and effective date for the stated campaign.
  • Basis: the specific instruction version and three sample library records.
  • Key: the Polish May catalogue, with reasons for both conditions.
  • Other options: the mistake each represents, without adding rules absent from the instruction.

Review the whole set for coverage. Five paraphrases of the May catalogue choice still address very similar cases. Add a different situation, such as a campaign after the July edition takes effect, if the instruction unambiguously determines the choice. Separately address a missing agreed language: the participant should recognise the need for more information in accordance with a confirmed rule. The set should distinguish decisions, not merely recipients' names.

This working record lets another person reconstruct the design of a particular exercise. A simple course does not need an elaborate grading system. It does need to let the next reviewer check whether the question concerns the right skill and whether its explanation has changed the company's rules.

Review the exercise together with the feedback

In a pilot, ask for a short explanation of the choice. Someone may select the May catalogue because they read the campaign date correctly, or because they always choose the first item. The selection is the same but offers different evidence of understanding. An explanation helps assess meaning; it does not reveal every thought or replace observation of the action.

You can ask for an explanation of the rule in the learner's own words (in Polish), followed by a short task in a test library. These are complementary checks. If a person knows the rule but cannot find the date in the interface, the tool's design may be the problem. A cognitive walkthrough of first use (in Polish) helps prepare that hypothesis before the session.

Review the feedback shown after an answer too. A model may correctly mark the May catalogue yet explain that “we always choose the oldest”. That comment teaches a wrong shortcut despite awarding the right point. Compare the explanation with the rule and case data. The July version is rejected because its effective date is after the campaign, not simply because the file is newer.

When answer order changes, follow the export into the platform. If the key is stored as a letter or number, establish how its association survives rearrangement. The learner's view should not accidentally retain a correctness marker from the author's material. Inspect the actual exercise and summary views, not just the input document.

For another language, check the question's meaning, answer descriptions and key explanation together. A field name should match what the participant sees in their own library. Translating only the question while leaving different terminology in the example can add difficulty absent from the original exercise. Keep each language version connected to the specific question and source instruction.

If a wrong answer can immediately be followed by another choice, a result after several attempts is not the same as the first choice. Retrying can be useful in an exercise for learning. Record what the displayed score means, however. Do not describe passing after a hint as independently identifying the right version on the first attempt.

An automated review by another model may help find gaps, but agreement with the generator does not confirm the key. When AI acts as a judge (in Polish), the judging itself needs assessment. For several decisive questions, return to the source and your own reasons. In particular, check questions the model accepted as correct, where an error may have remained unnoticed.

Collect the participant's impression separately if it can improve the material. A post-task difficulty and satisfaction survey (in Polish) records a report. “The exercise was easy” does not yet show that the person learned to choose a catalogue. It could also mean the form gave away the answer. Combine the rating with the observed process; do not use it instead of checking the skill.

Discuss an incorrect choice using the case conditions, without attributing a general inability to the participant. They may identify an unclear date or explain that they understood a field name differently. Record whether you are changing the material or returning to explain the rule. Then use a new case with different dates and names. Repeating the same question after revealing the key also shows knowledge of the answer; merely rearranging letters cannot separate that from applying the rule.

After a correction, retain the old and new question with the reason for change. Check whether only the wording changed or the basis for the key changed too. When updating the company instruction, find dependent questions and explanations. An old exercise may still work technically while teaching a rule that no longer matches the adopted process.

The scope of such a review can be discussed with Aeneia Audit. For Karda, the immediate outcome is an accepted exercise in which the catalogue choice can be justified by the language and campaign date. A trial with a new case will provide further information about applying the rule beyond a familiar example. A high score for spotting a generator's favourite word remains a score, but in a different contest.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear