Skip to content
Interface language: English
Audit / Signal

Long AI context: the document fits, but does the model retain the relevant exception?

Test exceptions, information position, source versions and combined conditions when AI works with long documents.

Article language: English

A woman holds a sheet of paper beside an ornate red cabinet of drawers.
In this article

The whole instruction fit into the conversation. The model acknowledged receipt, answered in a complete sentence and omitted the exception that changed the outcome. Everything went in, but the information decisive for the decision dropped out of the answer. A large context window can accommodate a document. It does not stamp it “read and understood”.

Context is the material available to the model when preparing an answer: instructions, document content, the earlier conversation and other supplied data. Its permitted size is a technical condition. Evaluating how the material is used requires a separate test, preferably one that checks a specific company task instead of admiring a number that fits on a slide.

The exception must change something

Imagine a fictional office room-booking instruction. The general rule allows rooms to be used until evening, but one room is unavailable for maintenance from 14:00 to 18:00. Someone asks to book that room at 16:00, when it is unavailable. A correct answer must combine the room name, time and exception. Repeating the general rule is not enough.

This makes a useful control because it needs no knowledge of a real office. Invent room names, dates and rules. Record the solution before running the test. The reviewer can then check reasoning against the material instead of guessing whether the model knows other rules from the internet.

Start with a question the document actually answers. If an exception applies only on a particular date, do not extend it to a whole month. If it concerns part of a room, keep that scope. An incorrectly prepared reference answer can produce a convincing report about a problem the model did not actually have.

For each question record which information is decisive and where it appears. If the answer requires three conditions, identify all three, including the one that excludes an apparently suitable room. This later separates finding a passage from correctly applying the full set of conditions. The reviewer need not invent the solution while reading the answer.

Also prepare a case without an exception and one missing necessary information. These three situations test different behaviour. The solution should use the stated condition, apply the general rule or identify missing data. Always demanding a confident answer can reward guessing precisely when the material does not justify it.

Ask for the passage supporting the answer. A page number or paragraph identifier must be checkable in the supplied version. Read the quotation too. A model may cite a real sentence about room availability that does not resolve the question asked. A tidy-looking citation does not repair an omitted condition.

In AI answer-quality assessment (in Polish), therefore, distinguish the correct result, the right basis and preservation of scope. These criteria can be reviewed separately. A correct answer reached by chance will not look the same as one justified from the material.

Before lengthening the material, run the same test on a short equivalent instruction containing every necessary condition. If the solution fails there, length need not be the problem. Check the question, meaning of the rule and assessment method. The short version provides a baseline for connecting any later deterioration to a specific change in the material.

Move the condition, not the document's meaning

Lost in the Middle, a 2023 study, found an effect of the position of relevant information on answer and retrieval performance in its tests. It concerned the configurations and tasks of that time. It suggests a way to test a current solution without predetermining how a model selected today will behave.

Prepare three versions of the same working instruction. Put the exception near the beginning in one, in the middle in another and near the end in the third. Preserve the other information. Move the condition with its necessary heading and references so the new position does not change its meaning. The test should compare use of the material, not three differently worded rulebooks.

Ask the same question in each version and record the answer. Check the result, basis and indication of uncertainty. If they change with the position of the exception, examine configuration, document preparation and the entire run. Do not assume a single cause immediately. First make sure the model received the material you intended to compare.

Keep a simple account of differences among versions. Someone repeating the test in a month should know that one condition moved and its wording was not also improved. If you simplify a sentence or add emphasis at the same time, the result concerns the combined change. That experiment may also be useful, but answers a different question.

Repeat selected tasks instead of saving only the most impressive result. Variation across AI responses with the same configuration (in Polish) helps determine whether a difficulty persists or appeared in one run. A variable result does not become stable because the best screenshot is placed in a presentation.

Position is only one test. A solution may find the condition in every location yet still combine it wrongly with a date or name. Keep the business task as the reference. Finding a character string is an easier success than preparing a correct answer for a particular case.

One retrieved fact cannot replace several connected facts

A room question may require combining the type of meeting, number of seats, time and room exclusion. When each appears in a different passage, citing one rule does not solve the task. Check the complete set of conditions needed for the answer. The reference should show which were used and their effect.

Aeneia conceptual photography

The current Gemini long-context guide distinguishes simple single-fact retrieval tests from tasks requiring multiple passages. It notes that accuracy depends on context and the number of items sought. That is a reason to measure your own tasks rather than transferring one demonstration result to an entire document workflow.

In a working test first ask for the number of seats alone. Then add a time condition, followed by a restriction on the type of meeting. Keep each result. Gradually increasing complexity helps show where the answer loses meaning. You can then investigate that part rather than changing the instruction at random.

Add a case where similar names refer to different objects. “Orion” and “Orion small” need not share the same rules. If the system confuses them, lengthening context may merely add material to the same misunderstanding. For data with identifiers, check how they are retained in the question and answer too.

In a separate test show two instruction versions with explicit effective dates. Ask about a particular date so a correct answer requires the right version. Do not call this a contradiction test if the documents simply describe different periods. The problem arises if the model combines passages from both into a rule neither document established.

If materials genuinely conflict, the team needs a resolution rule or a route to clarification. The model should not invent a document hierarchy. Before assessment, decide what it should do. Identifying an unresolved contradiction can be a correct result even when the screen does not show the expected “yes” or “no”.

Separate these tasks from assistant memory (in Polish). In this test you deliberately supply all necessary material in the current context. You are not yet asking what the tool retains between conversations. Mixing the mechanisms makes it harder to locate where information was lost.

Inspect what actually entered the conversation

A file-attachment button does not prove every part appeared in the material used by the model. Trace document reading, page selection and content preparation. If a table was flattened so that relationships between columns disappeared, the result may degrade before the main task. A large context limit cannot reconstruct that relationship.

The retained run should show which files and versions were used. Record intermediate steps too if the system selects passages or creates summaries. “We sent the whole catalogue” may describe the designer's intention. Acceptance requires the material actually used in that test.

Check the current configuration of the particular solution instead of transferring a limit from a presentation about an entire model family. Different modes of use can prepare documents and conversation differently. An application user may not see the same scope as a developer testing a direct call. Preserve that difference in the comparison so the result can be reproduced.

In a retrieval system, first establish whether the right passage was found. Only then assess its use in the answer. RAG and source freshness (in Polish) also covers the selection stage. The present test concerns a situation where the required information has already been supplied yet may still not be applied.

Pay attention to the previous conversation. An outdated assumption in it can make new material harder to interpret. In a comparative test specify whether the conversation starts empty or from a reproduced history. Otherwise the result also depends on things the operator happened to say a few minutes earlier.

For each error retain enough material to reproduce the case, respecting data-access rules. The answer alone will not show whether an exception was omitted, a page was lost or the wrong document version supplied. These causes need different fixes. Reducing them to “the model did not read” usually leads to a remedy that is too general.

Choose a way of working that fits the task

After testing, you may decide one large document set meets the need. Alternatively, split the work into finding the right material, evaluating the conditions and preparing an answer. If different agents perform the stages, review the actual handover of conditions and evidence between them, not just the last agent's answer. A guide to handovers between AI agents (in Polish) discusses that check. Compare both approaches using the same question and reference answer. Otherwise you are comparing different tasks under one name.

An excerpt that is too small can also be a problem. Removing the chapter with exceptions does not make a shorter answer better justified. Seek enough material for the task while retaining the condition's context. A summary should reduce unnecessary work, not cut away inconvenient passages that make a polished answer harder to produce.

Check automated answer evaluation separately. An AI model used as a judge may praise a fluent explanation even though both models omitted the same exception. The guide to an AI model as judge (in Polish) discusses how to assess the evaluator. Return to the source and prepared solution. A conversation between two confident systems still does not book the right room.

At acceptance describe the limits of use. A solution may answer simple parameter questions well but need checking when combining several documents. This distinction helps the team choose further verification. A general label such as “handles large files” does not tell an employee which task should make them pause.

Do not hide an unsupported category inside an overall sample average. If the solution handles individual parameters but omits time restrictions, show that difference alongside approved uses. The employee booking a room needs to know that this particular condition requires checking. A generally good score alone does not show a safe way to work.

For a conversation with Aeneia Audit, prepare one long document and a question in which a particular exception matters. Also bring the answer considered correct with its basis. This is a starting point for checking whether the system uses what it received, rather than merely confirming that everything was uploaded.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear