Skip to content
Interface language: English
Audit / Signal

AI data drift: how to check what changed in incoming questions

Compare periods, channels and case types to separate changing inputs from missing data, evaluation rules and answer quality.

Article language: English

A gardener adjusts the water flow at a prototype made from roots beneath a red greenhouse canopy.
In this article

A customer used to ask about one dimension. Now they send a description of an entire exhibition stand, three photographs and a question about whether everything will fit in the lift. According to the team, Rzeka's assistant still runs with the same settings, yet the people responsible for it are correcting more answers.

Rzeka is a fictional company that supplies equipment for events. It has added a portal for organisers alongside its existing form. The new channel brings different tasks, longer descriptions and attachments. The company wants to find out whether that change explains the problems. It should start by comparing the material and the results, rather than immediately replacing the model or restoring old settings.

Data drift is a change in the distribution of characteristics in incoming material, considered against a defined reference point. A practical review must name the characteristics being observed: question length, language, channel, task type or the presence of an attachment. The label "drift" alone does not say whether answers are worse or why. A change may be an expected consequence of expanding the service.

What are you comparing today's cases with?

Choose a reference period that answers the team's question. You could compare the current portal with the earlier way of handling enquiries, or examine change within the portal after its launch. Those are two different comparisons. Record the dates, channels and criteria for including cases before looking at the results. Do not choose a "good week" merely because it conveniently contrasts with the latest incident.

The Google Cloud Model Monitoring documentation distinguishes comparing production data with training data from observing changes in production data over time. The monitoring described applies to particular features and configurations. It does not mean that any assistant automatically understands the significance of every change in attachments. For a company review, first establish which reference point is available.

If you do not know the data a provider used to train the base model, do not present your own archive of questions as that training set. It is material from your process that can support a comparison between periods. At Rzeka, the question is how earlier simple enquiries differ from cases in the new channel. That difference can be described without access to the base model's training history.

Define the unit of analysis. One long case may contain a dozen or so messages, and one message may contain several questions. If you previously counted cases but now count each follow-up separately, part of the change will have been created by the report itself. Record how messages are grouped and how repeated submissions are treated. The data from both periods must use the same unit before their proportions can be compared.

Check whether the material is complete. Does the portal export contain the files, or only a note that files were added? Was a long description retained in full? A change in average length may come from a new field limit or a recording error. The guide to detecting anomalies and missing data helps distinguish a change in incoming material from a gap in observation. For more context, see the article on distinguishing anomalies from missing data (in Polish).

Agree on definitions before comparing. A "question with an attachment" could mean every message with a logo in its signature, or only a case that requires a file to be read. Agree on a rule and apply it to both periods. Otherwise the same message may be described differently even though the customer has not changed how they write.

Account for seasons and working days. The week before an event may bring urgent collection questions, while the preceding month brings requests for help choosing equipment. The comparison does not have to hide that difference, but it should name it. If the goal is to assess the portal, also look at a similar stage of preparation in the earlier period. Otherwise the effect of the new channel will be mixed with a change in tasks driven by the calendar.

Name the input change before attributing an effect to it

Choose a few features related to the assistant's task. For Rzeka, useful features include question type, number of issues in a case, attachment format and whether information must be extracted from a photograph. Add length and language if they affect handling. Do not collect every possible feature just because it can be put on a chart.

Show the number of cases in each group, not just the percentage. A change from one unusual message to three looks dramatic as a percentage, but still represents only a few cases. For rare cases, inspect the content and how it was handled. One serious error may deserve action even when it does not noticeably change the distribution of the whole dataset.

An automated comparison can flag a difference between the current distribution of a feature's values and its reference distribution. The alert threshold still needs its own justification. Do not copy a number from a documentation example onto a company dataset, or set it merely to make notifications disappear. Record the change you want to catch and who will investigate it.

Input change and quality change should have separate results. The Azure Machine Learning documentation distinguishes signals for data drift, prediction drift, data quality and model performance. For the last of these, it describes comparing outputs with ground-truth values. A different input distribution alone is no substitute for that assessment.

At Rzeka, more long questions might be good news: organisers have started sending complete descriptions instead of several fragments. If the assistant uses them correctly, there is no need to fix the change just because a chart differs from the old one. Similar message lengths do not rule out new difficulties either. Short compatibility questions may require different sources from the old questions about dimensions.

That is why you should examine the meaning of the tasks, not just text length. In a sample, you can distinguish questions about one feature, comparisons of several products and assessments of whether an entire set will work. If AI helps classify them, check its assignments on selected cases. Otherwise an error by the tool describing the input could be mistaken for a change in customer needs.

Aeneia conceptual photography

Look at combinations of features. The number of long descriptions and the number of attachments may each stay similar, while the two now occur together for the first time. That gives the assistant a different task: connect a written requirement with the right photograph. Choose a few such combinations based on the errors observed. A stable chart for one feature alone does not complete the review of the material.

Check the full processing setup

Confirm what really remained unchanged. The model name on the screen is only one part of the solution. Record the available version, instructions, knowledge sources, retrieval method, tools and how files are prepared and preprocessed. In the new channel, image recognition or description truncation may have changed even though the main model selection stayed the same.

Do not assume that all the increase in corrections comes from drift. At Rzeka, the portal might have started sending only the first page of a PDF. That error calls for a different fix from correctly transmitting new, harder cases. Prepare one reproducible example with missing information and check the material at successive stages. First establish what the model actually saw.

Also distinguish a change in the material from a change in evaluation rules. The company may previously have accepted an answer asking for clarification and now count it as a failure. Record the expectations used in both periods. If the criterion was made stricter, show the result under the new rule separately. Do not describe the whole difference as deterioration of the tool when part arose from the way it was evaluated.

When comparing results, retain a fixed sample of old tasks, as described in AI response regression tests. Add a separate set of new case types with independently established expected outcomes. The old sample shows behaviour on familiar material; the new set broadens the evaluation to current needs. Both are needed for this comparison. The related topic is explained in the article on regression tests for AI answers (in Polish).

Performance in each group may have stayed similar while the groups' shares changed. Consider an illustrative calculation only: simple cases have a ten percent error rate and complex cases thirty percent. With eighty simple and twenty complex cases, there are fourteen errors in one hundred cases. With a thirty to seventy split, there are twenty-four errors, even though the rates within the groups have not changed.

This is a fictional example, not a measurement of a tool. It shows why one aggregate result cannot establish whether the model became worse at a given task. Break down the results and shares of the groups. If the problem is concentrated in questions with photographs, identify that group and the type of mistake. A report saying "quality fell by this percentage" omits information needed to choose a fix.

Check evaluation delays too. Difficult cases may take longer to resolve, so quickly closed cases may dominate the evaluated portion of the most recent week. Comparing that portion with cases from a fully evaluated older month can distort the picture. State how many cases still have no result and when the evaluation will be completed. An unfinished case is not automatically either a success or a failure of the assistant.

Check whether the report includes every case sent to the solution or only those that received an answer. New files may be rejected before the model is called. In that situation, the material visible in its log may look stable even though customers are more often stopped earlier. Keep counts of inputs, rejections and actual answers, linked to the same unit of analysis.

Match the response to the difference you find

If data has stopped arriving, repair its transmission. If new questions require information the company did not previously make available to the assistant, check the scope of the sources and responsibility for the answer. If a task falls outside the tool's approved use, route it to a person. Retraining a model is one possible decision, not the meaning of every drift alert.

When the team chooses to change the model, assess it as a separate variant on comparable material. Comparing results during a model migration helps separate that intervention from the earlier input change. Do not announce that quality has been restored merely because the new model passed the old sample of simple questions. Also check the new tasks that prompted the review. This is discussed further in the article on comparing results when changing an AI model (in Polish).

When updating the reference baseline, keep the previous state. If the comparison baseline is moved automatically every time, a long-running change may stop triggering an alert even though the solution's original scope has not been reassessed. Decide who approves a new reference point and on what basis. A changed chart should not silently rewrite the acceptance criteria.

Check whether the AI's own descriptions and categories are feeding back into the inputs. Feedback from outputs can change the material without new customer behaviour. Separate the raw message from a summary previously produced by automation. Their connection is useful for diagnosis; treating them as two independent questions from the market is not. For the related method, consult the article on AI outputs feeding back into inputs (in Polish).

Assign an owner for responding to the signal and identify the material needed to close the review. Sending an alert does not confirm that anyone investigated it. Choose one case and follow it from the detected difference, through checking the sources, to a decision to fix or retain the solution. Record what remains unresolved too, especially when evaluated new cases are still missing.

After a fix, check the identified group again while retaining checks on the old tasks. Adding sources for complex projects could change answers to simple questions about a single product. Attach the solution version, scope of new examples and the case owners' determinations to the record. A decision to "keep watching" should have a date and a condition for returning to the evaluation, so it does not become a way to ignore a difficult group indefinitely.

Aeneia Audit can review a selected process and its material. Link the priority to the AI use-case register: a change in questions about a shade of colour calls for a different response from incorrect instructions for handling an order. At Rzeka, a useful review result should identify which new tasks the tool handles, where it needs correction and what data still needs to be collected. Further guidance is available in the article on prioritising AI use cases in a register (in Polish).

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear