Skip to content
Interface language: English
Audit / Signal

AI answer caching: when a shortcut repeats an old mistake

Check when AI answers are reused, how source changes and permissions affect them, and whether a refresh reaches the user.

Article language: English

Illustration: part of a stone column, with a thin line marking a crack.
In this article

The assistant replied instantly. It gave the conditions for the previous workshop, even though the company had changed them that morning. The answer was polite, complete and entirely out of date. First check whether the model generated a new answer at all. The system may have saved work by returning an earlier result.

A cache lets a system reuse data or part of a calculation, but the name covers different mechanisms. Storing a finished answer calls for different rules from reusing a processed part of a prompt. If both are simply called "AI memory", it becomes difficult to know what needs refreshing.

Ask what was saved

Amazon Bedrock's prompt caching documentation describes reuse of eligible prefixes, the beginning of a context, for supported models and APIs. Availability and conditions depend on the configuration. This concerns processing the input, rather than simply returning an earlier answer to the user.

A separate application layer may save a question and the finished result, then return it for a later request without generating it again. That is the case considered here. The company needs to know where the decision to reuse a result is made and which conditions are compared. The word "cache" on an architecture diagram does not answer either question.

Imagine a fictional studio running illustration workshops. On Monday its assistant answered which materials participants needed for one session. On Tuesday the organiser changed the venue and the supplies provided. Monday's answer may still look credible because the workshop name remains correct, while the details that needed updating are easy to miss.

First reconstruct the execution trace. Did the answer come from an application cache, a new model run or somewhere else? Which material was available when it was created? When was it saved, and under what rule was it considered current? Without that evidence, a team can spend time rewriting the model's instructions when the model was never called.

Distinguish this from assistant memory between tasks. A remembered preference can influence a new run. A finished answer, by contrast, may be returned without checking a changed source. Both need attention, but they do not necessarily share an owner or a delete button. The Polish article on assistant memory (in Polish) explores the distinction.

The same question can refer to a different state

"What should I bring to the workshop?" sounds identical before and after a programme change. The cache key, the information used to find a saved result, should reflect differences that matter to the answer. In the fictional studio these could include the session and the version of the approved instructions. The question alone does not establish those conditions.

Do not make a speculative list of fields. Trace what the result actually depends on. If an answer depends on the venue, leaving it out of the comparison can combine different cases. If it does not depend on the order of characters in a technical request identifier, adding that identifier to the key can prevent useful reuse. The rule has to fit the task.

Look at context supplied by the application. A user may not name the event because they opened the assistant on its page. Does that context enter the comparison? The same short question on two pages is not automatically the same task. The interface often knows something that does not appear in the chat field.

When an answer uses several documents, establish what their combined version means. A changed attachment may affect only some results or the entire set. Decide how to identify dependencies and invalidate the right entries. Clearing everything may be simpler, but it too should be a deliberate decision whose effect on the service is checked.

In a RAG workflow, updating the source collection does not necessarily remove completed answers stored in another layer. Check how the two operations connect. A new model run retrieving the updated document correctly does not resolve the whole problem if users still receive an older saved answer. See the Polish guide to AI search sources and freshness (in Polish) for the source-checking context.

Similar questions can differ in a decisive way

A semantic cache looks for saved answers by similarity rather than only exact character matches. Microsoft describes its llm-semantic-cache-lookup policy, which uses vector proximity and a matching threshold. It also warns that an answer based on similarity may be wrong or outdated for the current request. Test the threshold for your own use.

In the workshop example, "What do you provide on site?" and "What do you not provide on site?" concern the same subject but need different answers. Prepare pairs like these and observe the whole application. Do not assume every method will confuse them; check whether your configuration preserves the difference and which saved result it uses.

Likewise, test a question about a different session, or the venue alone without the materials list. An answer may share many sentences with an earlier result and still need a separate condition checked. It is not enough that the text concerns the right event. It must fit the particular question and current information.

Aeneia conceptual photography

Embeddings and similarity help explain how a candidate is chosen. Return to the operational decision: when may a finished result be returned without another check? A useful suggestion for a similar document should not automatically become an answer. Those are different uses of the same result list. The Polish guide to embeddings and similarity (in Polish) discusses the limits of similarity.

Access to an old result needs a current decision

The organiser and a participant may ask the same question but be entitled to different information. Check whether entries are separated according to relevant permissions and whether those permissions are reassessed when the result is read. Permission to receive an answer on Monday does not establish access after a role change on Tuesday.

The Microsoft documentation identifies vary-by as a way to partition entries, including by user or group. The existence of that option does not prove a particular configuration is correct. Test with two accounts and again after access is revoked. Check the answer each user sees, not only the rule saved in a control panel.

Do not place information intended for a narrower group in a public answer on the assumption that the cache will somehow recognise the boundary later. Establish which material may be used to generate a result and who may receive it. Where several services take part in retrieval, find out where the check actually happens.

Connect these tests to AI integration permissions. Restricting a new source retrieval may not be enough if the application still returns an earlier answer. Test both paths: creating a new answer and replaying an existing one. A user need not know the difference, but the system must handle it. Related access checks are described in the Polish article on AI integration permissions (in Polish).

A saved refusal can become outdated too

In the fictional studio the materials list was not ready when a participant first asked. The assistant correctly said it could not confirm the list. If the application saves that answer, it may keep reporting the earlier absence after the list is published. The old text was honest, but the new information has been missed.

Check whether the system also saves refusals, empty results and answers produced after a source error. Establish when each may be reused. A temporary retrieval failure should not become a lasting statement that the organiser has published nothing without another check. A technical event and the state of knowledge mean different things to the participant.

Test after a new document appears, not only after an existing one changes. If an entry is tied solely to the version of an older source, the application may not notice that the collection has grown. Find out how it detects that change. The person updating information should know whether publishing the file is enough or another action is required.

Keep the sequence in the test: first question, justified refusal, new material, repeated question. A screenshot of the final answer alone will not reveal whether the issue was retrieval, storage or the publication procedure. Record the case identifier, read time and identified source version. These connect visible text to the right execution without copying the whole conversation into every report. The history should allow the change to be reconstructed without guessing what was available to the model. If updates arrive late, also examine late statuses and case versions, so an old signal cannot restore an outdated state. The Polish guide to late events and status (in Polish) considers changes arriving out of order.

When an outdated entry is removed from one layer, check the other places where it is presented. A browser or intermediary service may still hold the old answer. Test through the user's ordinary route and compare what they actually receive. A successful clearing operation in an admin panel confirms that operation, not the removal of every possible copy.

See the refresh in the answer

Establish which event invalidates an entry, who can initiate it and how success is recognised. A change to a source, instruction, offer or permission may be relevant. Time passing alone does not always reflect changes in the work. A new condition may take effect well before the old result expires.

In the fictional studio, make a controlled change to the workshop instructions and ask the earlier question again. Check the answer through the normal user entry point. Technical acknowledgement of a request does not establish which conditions appeared in the response. If a refresh requires restarting the application or manually removing an entry, include that in the procedure. Do not describe an automatic action that the test did not confirm.

Also measure behaviour when there is no entry or the cache fails. Agree on the next path, such as a new run or a clear unavailable message. If the system switches to another model, examine AI task routing. Bypassing one layer must not accidentally bypass data or access conditions.

Choose retention time according to the importance of the information and how it is updated. Where material changes after an organiser's decision, even a short expiry does not guarantee correctness at every moment. Agree on an acceptable delay and handle the point of change. The person responsible for the service needs to understand that operational decision.

Compare savings with the cost of completing the case. Fewer new calls may reduce one line of the bill while increasing corrections and support questions. In the test, retain the result's source, response time and corrections needed. That shows whether reuse helps with the task or only looks good in a call counter.

For an Aeneia Audit discussion, prepare the question, the earlier result, the current document and the time of the change. These give the team a basis for examining what was saved and why it was considered useful. A fast answer saves work only if it also helps its recipient. Otherwise the company is simply delivering something to correct more quickly.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear