AI in shadow mode: compare the result without repeating the action
How to compare a new AI variant on copies of current requests, prevent duplicate actions and interpret the trial within its actual scope.
Article language: English

In this article
Imagine an assistant whose new version identifies cases correctly in last year's test set. The company has since begun selling a different service, customers send longer attachments, and questions arrive through a new form. The historical test still matters, but it is becoming less representative of today's work. You can give the new variant copies of current requests and examine its responses while the existing version continues serving customers. That is shadow mode. The shadow should observe, though, not handle a case for a second time.
The most important decision comes before choosing a tool for copying traffic: exactly what may the second run do? Two sent messages, two tickets or two bookings would turn a model comparison into a very practical comparison of apologies. You need a boundary between calculating a proposed response and carrying out its effects, plus evidence that the boundary actually works.
What does a copy of a current request provide?
The documentation on production and shadow variants in SageMaker AI describes copying some requests to the observed variant. Its responses are logged for comparison but are not returned to the caller. This separates the response used by the customer from the one assessed during the trial.
First write down the research question. For example: does the new variant correctly recognise current requests to change a service's scope, including those sent through the new form? "We are testing the new model in production" is too broad. It does not say which differences to collect, who will assess them or what the company will learn after a week. The observation period should follow the kinds of cases and their arrival patterns, not just an open slot in the team's calendar.
Distinguish this trial from preparing an isolated test environment for AI automation (in Polish). There you can deliberately produce rare errors, change the clock or submit a fabricated attachment. Current traffic instead shows cases that actually arrived during a given period. It will not automatically supply an emergency if none happened then. The checks complement each other: a successful week of observation does not replace deliberate tests.
The choice of requests to copy is also part of the result. Only short messages without attachments would make comparison convenient but reveal little about the new form. Record which case types you include, which you exclude and why. If some material cannot go to the new provider or environment, exclude it before copying. Carry that limitation through to the conclusion instead of hiding it in technical settings. The idea of observation alone does not authorise connecting a copy to the live service. An agreed scope and separate approval to run the trial are needed.
Where does observation end and a second action begin?
Azure Machine Learning distinguishes splitting traffic from mirroring it. In mirroring, the second deployment's results are not returned to customers, although metrics and logs are produced. Not returning the response does not by itself guarantee that an application behind the connection will take no action in other systems.
Consider a working example. The existing assistant reads a request to move an appointment and prepares a proposed response. The new version can also change the date in a calendar. In the trial you want to assess whether it proposes the right date without allowing it to book meetings. Disconnecting the chat window is not enough if the agent still has a calendar tool and write permission. A model instruction saying "observation only" is not proof that the action has been disconnected either.
Agree with the implementer to replace such tools with controlled counterparts that record intent. Instead of sending a message, the second run creates a record: "I would send a message with this content to this role." Instead of changing an appointment, it records the proposed date and rationale. If assessment requires the calendar's state, agree separately how it may be read. Read access and permission to change data should not accidentally travel in the same permission package.
The documentation on traffic mirroring in Istio says mirrored responses are discarded. Discarding a response at the traffic layer does not, however, control every effect of code handling the request. Acceptance should therefore check prohibited application outputs, not merely confirm that the customer sees the old response.
Before using current data, attempt a prohibited action on a synthetic case and inspect the result in the destination system. The record should show what was attempted, where it was blocked and whether any record remained. This is a proposed acceptance condition, not a claim that the described example underwent such a test. If the block cannot be demonstrated, return to an isolated comparison. Revisit the live-traffic trial after removing that obstacle and obtaining separate approval to start it.
Blocking an action also changes the agent's subsequent interaction with tools. When a calendar substitute records an intent to book, it should return an understandable trial status instead of pretending that a real booking exists. Otherwise, the model may generate confirmation of a nonexistent meeting, and that text could be assessed as the supposed result of the whole process. Agree how far the response remains comparable with the active path. Mark later steps that depend on the changed state as simulated, or end the observed run before them. Avoiding a second action is desirable, but it limits what the result can support.
A trial record should separate three facts: what the model proposed, what the control mechanism allowed and what actually happened. Do not merge them into a single "success" status. In the example, a correct date can coexist with a correct refusal to make a booking. An assessor should recognise both without searching through many unrelated logs. The distinction also helps find a case where a block took effect only after an unwanted partial write.

How can two responses to one case be compared?
Start with a pair of results linked to the same input. Give the pair an identifier and record both variant versions, input time, case type and completion status of each run. A missing response from the new model must remain a missing response. If pairs with errors or timeouts disappear from the comparison, the quality assessment will cover a more convenient subset than the actual trial.
In the calendar example, the state of information matters too. The old variant may have seen an available slot while the new one read the calendar after that slot was booked. A difference in proposals would not necessarily mean a difference in reasoning. Decide which data share a version, which are fetched independently and how a state change between calls will be marked. If you cannot determine this, retain the case as non-comparable with the reason recorded.
Assessment is not just a question of identical sentences. Two different responses can both correctly identify the same date and need for customer consent. Two similar ones can both omit a restriction in an attachment. Prepare criteria tied to the decision: recognition of intent, data used, proposed action, the condition for carrying it out and information for the person involved. Assessing agreement with the old model alone would treat each of its errors as a standard to copy.
Each divergent case needs someone able to read it and justify a decision. You can hide which variant produced a response during assessment and compare the results afterwards. Separately retain disputes requiring the process owner's knowledge. This is more useful than an automatic "new is better" label that nobody can explain on a particular example.
Does observation burden current work?
A copied request means another run, which must be included in the trial's resources and costs. Agree an independent way to stop the shadow and a condition for stopping observation if it begins to interfere with service. Do not make the customer wait for both variants merely because that makes pairing the report easier. The comparison team can wait for a result; the customer did not sign up for an organisational experiment.
If both runs share a provider or budget, account for the dependency described in AI API limits, queues and shared budgets (in Polish). Record how many requests were selected, actually copied, completed and omitted. Without that, a low number of detected errors can easily be confused with a low number of cases that reached the second variant at all.
Selecting part of the traffic should not silently remove more difficult inputs. If the mechanism stops copying long messages under load, record that rule and the cases in which it was applied. The result will then be honest for the observed scope. It will not answer how the new version handles all long messages during peak traffic.
There is also another flow of data: stored input, the shadow response, assessors' rationales and tool logs. Update the AI data-flow register (in Polish) with recipients, access scope and the arrangements for ending storage of the trial material. A copy does not become anonymous because it is used for research. The specific data scope must be decided before starting; this article does not replace a legal assessment of its use.
What should the trial conclude with, and what remains unknown?
The report should tie results to the limits of observation. Give the period, selected case types, both solution versions, completeness of pairs, and resolved and unresolved differences. Show a few real contexts within the recipient's permitted scope. Also preserve the rule used to select examples: if you discuss only the largest successes or only failures, the reader should know. Attach the status of each discrepancy and the owner of the next step. An unresolved case does not become correct because it did not reach the closing meeting. Agreement percentage alone cannot explain whether differences concerned response style, an appointment date or a proposed change to customer data.
Shadow mode cannot show the full user reaction to a new answer when users did not receive it. Nor does it prove how an entire chain of actions will proceed once tools are enabled. That requires separate conditions for the next stage. For integrations across systems, it helps to map partial execution of agent tasks (in Polish), including what remains to be clarified after work stops.
A decision to change versions belongs to a separate acceptance of AI model migration and result comparison (in Polish). Decide which shadow findings are sufficient, what needs improvement and which questions remain open. A trial can end without switching while preserving a valuable set of current cases. That is a better result than deployment merely because the scheduled observation period has elapsed.
Aeneia Audit can help organise this operational assessment: the trial question, action boundaries, evidence of execution and responsibility for the decision. Its scope must be agreed according to the process used and access to materials.
Primary sources were checked on 30 October 2026. The providers' examples illustrate the trial mechanism; they do not mean every integration uses the same configuration or has ready-made action blocks.







