Skip to content
Interface language: English
Audit / Signal

Audit an AI search tool: sources, freshness and missing answers

Check whether an AI search tool finds the right sources, preserves their meaning and respects access after documents change.

Article language: English

Illustration: part of a stone column, with a thin line marking a crack.
In this article

An answer with three citations can look like a meticulously completed assignment. If all three lead to an outdated instruction, the care may be mostly typographical. Check a company's AI search in two steps: did it find the right source, and did it answer faithfully on that basis? Then examine freshness and access. A link helps you verify an answer; its presence alone does not establish that the system respected the source's conditions and scope.

RAG, or retrieval-augmented generation, combines finding material with a model's preparation of an answer. Microsoft describes the pattern in the Azure AI Search documentation. An audit does not have to begin with every technical setting. Start with employees' questions, the documents currently in force and the intended boundaries of an answer. Then investigate which stage accounts for a discrepancy.

Choose questions for which the company knows the basis of an answer

Collect questions from actual work: how to prepare a particular document, who approves an exception, which form applies or what a service covers. For each one, identify the current source and someone who can confirm its meaning. Do not start with questions chosen simply because they sound good in a product demonstration.

In a hypothetical service company, an employee asks who takes an urgent request outside the usual queue. The collection contains a current instruction, its previous version and meeting notes in which a change was discussed. A correct answer should distinguish an established rule from an idea. The number of documents does not decide which one takes precedence.

Record the conditions that matter to each question: the user's role, the type of case and the time it concerns. If the answer depends on those details, the system should ask for them or state the limitation. Do not score it solely on whether a few words match a model answer. The person relying on the process needs the right decision in a particular context.

If the company cannot identify an authoritative source, record that gap before continuing the test. Replacing the model will not put conflicting instructions in order. A chat interface may give the company library a voice, but eloquence alone will not organise its catalogue. A data flow register helps establish where materials enter the tool and who is responsible for keeping them current. For more context, see the article on mapping AI data flows (in Polish).

Check retrieval before judging the answer's style

For each question, examine the passages the system retrieved. Do they concern the right service, version and situation? Do they contain a condition needed for the answer? If the source is wrong or incomplete, polishing the model's language will not remove the cause. First fix the route to the material.

Prepare your own test with three ways of asking the same question: the term used in the instruction, the everyday expression employees use and a short description of the problem. Record which versions lead to the right source. This does not prove performance across an entire vocabulary; it helps reveal a mismatch between documentation and the language of daily work.

Also test a question containing a name used by more than one department. An abbreviation may mean one thing in customer service and another in a project team. The answer should account for the context or ask for clarification. Selecting the most similar passage does not necessarily mean the task has been understood correctly.

For scans, examine an earlier stage: OCR before AI analysis. A misread name or table can impair retrieval even when the text-comparison mechanism follows its configuration. An audit should separate those causes instead of attributing every mistake to one model. The related topic is explained in the article on checking OCR before AI analysis (in Polish).

Open the citation and check what it actually supports

A link should lead to material that lets you verify the claim. Open it from the user's account and find the relevant passage. A similar document title is not enough. If the answer makes several distinct claims, check the basis of each important element, particularly deadlines, exceptions and responsible people.

In the hypothetical example, the instruction says that a coordinator assigns the case after an initial assessment. The AI's answer should not turn that into a promise that a particular specialist accepts it automatically. The document can be the right one while the conclusion is still too broad. That error belongs to the answer stage, not retrieval itself.

Record cases in which a citation is genuine but does not cover the whole statement. Check whether the system labels its own proposal as a proposal and can identify a missing basis. A lengthy comment after every sentence is unnecessary; the distinction matters where someone needs it to decide what to do next.

Agree answer verification rules with the team. A user should know when a short answer is enough for orientation and when a source check or a decision is needed before an action. Trust should fit the task, not the elegance of a citation. This is discussed further in the article on training a team to verify AI answers (in Polish).

Test a document change and the withdrawal of an old version

Add an update to one important instruction to the test. Record when the source changed, how the change reached the search system and what the next answer contained. Do not assume that changing a file immediately changes every copy the system uses. Check the actual configuration and refresh path. If the process stores completed answers, also examine the AI cache after a source change (in Polish), so a refreshed document does not leave an old result on another path.

Then withdraw a document in a controlled environment. Does it stop appearing in retrieval? Can an old link in a saved conversation still show what an earlier answer was based on? Current availability and the ability to explain a historical result are different needs; decide how to reconcile them.

Set retention and deletion rules for the search collection and any other copies that actually exist. The guide to AI data retention and deletion (in Polish) explains how to define and check the scope. Do not record a single declaration of deletion when nobody knows which stage was checked. The record should identify the scope and the evidence.

After a significant source change, return to regression tests. An update may improve one answer while changing another that used the same passage. Keep a small set of important questions instead of beginning acceptance with a random conversation each time. Also check for a change in the kinds of questions reaching the system; an old set may no longer reflect daily work. For the related method, consult the article on regression tests for AI answers (in Polish). Further guidance is available in the article on data drift and changing questions (in Polish).

Test access in the result, not just in a declaration

Use at least two roles with different permissions for controlled materials. Test retrieval, the answer and opening the citation. It is not enough that a person without access cannot open a document if the tool has already revealed its content in a summary. Both the result and the source must respect the agreed scope.

Aeneia conceptual photography

Microsoft describes a pattern for filtering results by permissions. Its documentation distinguishes the filter itself from authentication and authorisation. An audit therefore needs to examine the complete mechanism in the application, rather than treating a field name or setting as proof of protection.

Add a test after access is removed and after a group changes. Record when the change takes effect and what remains in an earlier conversation. Do not assume every product behaves the same way. An integration permissions review guide (in Polish) helps match the test to the connection actually in use.

Material retrieved from documents should not become a new instruction controlling the application. A separate test of that boundary is described in prompt injection in documents (in Polish). Even a correctly retrieved file remains a source to assess; it does not own the agent's permissions.

Test the conversation after the first question

The first answer may be correct while the next question inherits the wrong context. In the hypothetical company, an employee first asks about a standard service, then about an exception for another team. If the system keeps using the first material without recognising the change, it can produce a plausible answer to a task other than the one now being asked.

Prepare a short sequence: the initial question, a clarification and a clear change of subject. Record the expected source for each step. Check whether the system retrieves the necessary material again or clearly says it is relying on the earlier answer. Do not assume conversation memory always improves the result.

Add a user correction. If an employee says the question concerns a different unit or period, the tool should use that information in the next step. Agreeing politely is not enough if the answer still comes from the wrong instruction. Assess the material and the conclusion, not the courtesy of the first sentence.

In the report, distinguish a retrieval error from an error in maintaining context. That helps identify whether the source, the way the conversation proceeds or the presentation of limitations needs to change. A user should know when they are starting a new task and when they are clarifying an earlier one. The interface can make that distinction easier rather than hiding every issue in one long chat.

Separate lack of access from information about the document

When testing different roles, check titles, short descriptions and suggested follow-up questions as well. A result may withhold the full document and still display information the user should not receive. The assessment should cover what the application actually returns, not only whether a file opens after a click.

Prepare controlled material with clearly marked fictional content and restricted access. Ask about it from an authorised role and an unauthorised one. Record the answer, the source list and the suggestions. Repeat the same checks after changing permissions. This tests the system without using real data that needs protection.

Do not demand an excessively detailed explanation of a refusal if that explanation would itself reveal material outside the user's scope. Agree with the owner how to communicate the absence of an answer. Sometimes it is enough to say that no supporting material was found in the accessible collection. That decision should fit the company's rules and the tool's actual behaviour.

Check conversation export and sharing too. Correct access at the time of a question does not determine who later receives a copy of the answer. There may be no need to rebuild the entire workflow at once, but identify this boundary and assign responsibility for onward distribution.

Score sources and answers in separate columns

For every question, record whether the right material was found, whether the answer preserved its meaning and whether the user was entitled to see the result. Add the type of gap and the time needed to check it. You do not need a single combined score that hides an important error beneath a good style rating.

For example, an answer may score well for finding the right source yet need correction because it omitted an exception. In another case, the source is outdated even though the summary represents it faithfully. These observations lead to different repair tasks. That is what an audit should enable, rather than leaving only a general impression that the chat sometimes gets things wrong.

Include unresolved cases and questions outside scope in the decision record. If the collection covers instructions for one service, do not claim acceptance of the company's entire knowledge base on that evidence. The owner should be able to identify uses approved for work and those needing another test or a different source.

Agree how to handle missing answers and assign corrections

Prepare a question with no basis in the collection, a contradictory question and one the document answers only partly. A correct result may be to identify the gap, ask for clarification or refer the person to the right owner. Do not treat an honest admission of a gap as an error when the alternative is an invented condition. Company rules should not appear just because the model felt inspired.

Record the cause separately: missing document, wrong retrieval, outdated version, incorrect conclusion or access restriction. Each category leads to a different correction. The owner of the material should not be asked to rebuild a model when the actual task is to approve the instruction in force.

If AI meeting notes feed the collection, label their status. The guide to checking AI meeting notes (in Polish) describes checks before those notes are used further. A working summary should not replace an approved decision without a check. Search needs the source hierarchy the company actually uses, not merely a larger pile of files.

Before acceptance, ask a second person to check several answers without commentary from the person who configured the system. They should find the source, identify an important condition and recognise a missing basis where a gap was deliberately included. If they judge only the credible tone, improve the verification instructions. The interface can make checking easier, but it should not suggest that every text with a link is already an approved company answer.

Also record on the acceptance card what the user does after finding an error: whom they notify, which passage they point to and whether they can safely continue. A thumbs-up or thumbs-down rating alone does not guarantee that the issue reaches someone able to correct the source or the answer mechanism. Test one report from beginning to end.

Before purchase or expansion, use the guide to assessing an AI vendor (in Polish) to check these conditions. To run the test against your own questions, see the scope of Aeneia Audit.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear