AI logs without excess data: how do you keep evidence of what happened?
Design an AI process event log that helps explain failures, costs and operations without automatically copying confidential conversations.
Article language: English

In this article
“Success” in an event log sounds reassuring. But what succeeded: receiving a model response, saving a draft or sending a message? A useful log lets you reconstruct an important step, connect retries and investigate an error. It does not require copying a customer's entire document into another database. Record the evidence needed for a decision. A work log does not have to become a diary of every piece of company data.
The approach below concerns a small process within a business. Choose the detailed fields, retention periods and access rules for your own task and the infrastructure you actually use.
What questions should the log answer?
Start with five questions to ask after a task fails. Who started it? Which version of the process was running? At which step did it stop? Was there an external effect? Who needs to act next? If a proposed field helps answer none of the agreed questions, it needs further justification before being included in the default log.
A support employee usually needs the case status and a safe way to resume work. The person maintaining the integration needs an error code and a call identifier. The person responsible for billing needs usage information and a way to attribute the cost to the process. Do not assume all three roles should see the customer's entire conversation. Design the views around their needs, not the convenience of an administrative panel.
Agree on what “success” means too. For text generation it may mean that a response was received. Sending an invoice is a separate step and needs its own status. Our guide to an automation process audit (in Polish) examines boundaries like these. The log should reflect the actual steps so that one green status cannot conceal an unfinished task.
During a workshop, prepare an example query: “Show cases where a draft was created but the save in the destination system was not confirmed.” If answering it requires manually searching hundreds of conversations, the status information lacks structure. Adding the full contents will not fix that gap; it will only increase the amount to review.
How do you build a minimal event record?
For your own process, start with the time, task identifier, step name, configuration version and result. Add an identifier for the related operation and distinguish the first attempt from a retry. The values should come from the application performing the action. The model's free-form description may provide context, but it should not by itself mark an operation as completed. When processing many cases at once, check how you match a batch result to its record (in Polish), including when responses arrive in a different order from the inputs. With several event sources, examine late statuses and the actual order of changes (in Polish) so that log order does not masquerade as the case's history.
A task identifier can connect events without copying their contents. Do not put a customer's name, contract number or email address in it. Choose a technical identifier and a controlled mapping to the case in the system that already manages that information. If the identifier can link a record to a particular person, still account for that in your data protection rules.
The OWASP logging guidance stresses choosing events to fit their purpose and handling tokens, passwords and confidential data carefully. It also describes the need to protect logs against unauthorised access and modification. For this project, that leads to a concrete acceptance task: inspect what the logging mechanism actually records instead of relying on settings alone.
Result codes should be stable. Rather than ten versions of “something went wrong”, agree on a small set: access denied, limit exceeded, invalid input, no confirmation or provider failure. You can make the full technical message available through a separate, restricted diagnostic mode if it is needed and has been checked for sensitive data.
When is conversation content needed?
Sometimes a quality problem can only be explained by comparing source material with the response. That does not mean every conversation needs to be stored indefinitely. Design a separate diagnostic path: a named case, a justified purpose, an approved scope, defined access and a deletion date. The ordinary operational log can remain limited to events.
Look at intermediary tools as well. Your application may not save prompts, while an observability platform captures complete requests. A library may put document contents into exception details. Add diagnostic locations, exports and notifications to your own data-flow register. Our guide to an AI data-flow register (in Polish) explains how to prepare it. Every additional copy should have an owner.
Use a simple order. Begin with operational metadata, then reproduce the issue with synthetic test data. Only if those steps do not explain the problem should you consider inspecting a limited amount of real material under the agreed procedure. This sequence reduces the data copied reflexively during each failure and makes the case easier to close afterwards.
Do not call masking anonymisation without checking the result. Removing a person's name from a question may still leave their job title, a small town and the exact date of an event. For diagnosis, you can often replace the whole situation with a synthetic equivalent. Keep the feature that triggers the error, such as an unusual date format, without retaining the person's biography.
What might a duplicate-ticket investigation look like?
Suppose, hypothetically, that an assistant helps create service tickets. After a delay, an employee retries the task and two records appear. The existing log contains two model responses and a shared “done” status, but no operation identifier from the service system. The team cannot tell whether the duplicate came from the model, the application or the user's retry.

The new design records the start of the task, the proposed record, employee approval, the save attempt and the confirmed ticket identifier as separate events. A retry remains linked to the original case. If no response arrives, the system does not automatically assume the save failed. It marks an uncertain state that must be checked in the destination system before another attempt.
In a proposed controlled test, a simulated service system accepts the record but delays confirmation. Acceptance checks whether the application shows the right message and whether the operator can find the result without creating it again. Only then can the team see whether the log helps them resume safely. The presence of every field alone does not establish usefulness.
The proposed report would contain the test ticket identifiers, the sequence of events and a recommended change to retry handling. Customer names and the full fault description are unnecessary. The conclusion would apply to this particular save path; it would not establish that other integrations behave the same way. Each may confirm an accepted operation differently.
How should you set access and retention?
The retention period should reflect the time needed to detect and investigate a problem, the organisation's obligations and the data category. Do not copy a default number of days from another project. Event records, detailed diagnostics and an aggregate report may have different justifications. Your own retention and deletion plan should also cover copies and exports. Our guide to AI data retention and deletion (in Polish) discusses the plan.
Separate permission to view case status from the ability to download logs in bulk. An export creates a new flow of data and often bypasses restrictions in the panel. Name an owner for the export, where it will be stored and how it will be deleted. The process owner should know whether evidence of actions will be available during an incident and who can obtain it.
Practise what happens when an administrator leaves. Documentation of the schema and filtering rules cannot live only in that person's private notes. A successor should understand the fields and distinguish a missing event from confirmed evidence that an operation did not occur. During a failure, that distinction can determine whether a task is resumed safely.
If you suspect that a log contains confidential data disclosed to the wrong person, follow your organisation’s incident procedure. Our guide to an AI data incident (in Polish) discusses the first response. Do not rush to delete the entire collection before establishing evidentiary needs and responsibility. The operational response needs to coordinate access restriction, preservation of facts and assessment of further obligations.
What if some events are missing?
A missing entry does not necessarily mean the operation did not happen. The log may have been unavailable, the application may have stopped before writing, or the event may have gone to another tool. In the diagnostic instructions, specify which independent source can confirm the state. For a created ticket that will usually be the destination system; for a saved file, the relevant repository and its change history.
Introduce an uncertain outcome instead of forcing a choice between success and failure. The operator should know which place to check before retrying. That state must have an owner, a review deadline and an escalation route if available sources cannot establish whether the action was performed.
Design an alert for interruption of logging itself. It need not contain the task's contents. Identifying the failed component, detection time and person responsible for responding is enough. Check that an overall application health status does not hide missing logs. A process can keep running while losing the ability to explain its effects later.
In an acceptance exercise, deliberately disable a test event receiver. Observe the application's behaviour and its message to the operator. The team should decide in advance which operations may continue with an incomplete process record and which must stop. The decision depends on the significance of the action and the availability of other confirmations, so it should not be left to a library's incidental behaviour.
When designing identifiers, connect the log with duplicate and retry handling (in Polish). Events should make one intended operation recognisable across several transport attempts. Without that link, several log entries can look like several independent tasks.
How do you check the log before acceptance?
Acceptance covers both the system's behaviour and its ability to answer the earlier questions. Choose several synthetic tasks: success, denied access, interruption after partial completion and a retry. Ask someone who did not design the logging to reconstruct what happened. Their difficulties often expose gaps better than another field-by-field checklist.
Insert recognisable fictional markers resembling a secret and a customer name into test data. Check for them in the application, monitoring tools, alert messages and exports. This tests what is being recorded; it is not an examination of real credentials. Repeat the test after a configuration change, because a filter may behave differently for an ordinary event and an exception.
Diagnostic work is also part of the total cost of your AI process. Our guide to total AI process cost (in Polish) explains what to include. Account for storage, data transfer, alert review and the time needed to explain a failure. A cheap log that nobody can interpret may increase the cost of every incident. An overly detailed record, meanwhile, creates additional work to protect and sort the data.
- Every recorded field has a purpose and an owner.
- Model success is separate from the outcome of the destination operation.
- Identifiers connect steps and retries without copying case contents.
- Ordinary events, exceptions, alerts and exports have been checked.
- Diagnostic access has a defined scope and duration.
- An operator reconstructed a synthetic problem from the available events.
- The team knows what to do if the log itself is unavailable.
If a process works but its outcome is hard to explain, Aeneia Audit can help define the evidence needed.









