AI pilots at work: which evaluation criteria should you set before testing?
Plan a task-based AI pilot with test cases, reference results, error criteria, review costs and conditions for further use.
Article language: English

In this article
An AI pilot should answer a question about the company's work, not end with “Impressive, isn't it?” Choose a task, prepare cases and set the criteria before seeing the results. Measure both the usefulness of the output and the work needed to check it. Otherwise a demonstration can win a contest whose rules were written after the winner was announced.
Set access boundaries and a way to stop the test in advance. Assessing usefulness means looking at output quality, human effort and the conditions for processing the material. A count of tool runs shows only part of the picture.
How do you choose a task for the first trial?
Look for a repeatable activity with recognisable inputs and outputs. The team should be able to say what correct work looks like. A useful question might be: can the tool help prepare a draft summary of an approved document that a person can check efficiently?
Describe how the work is done now. Who prepares the material, how many stages does it go through and where are corrections most often needed? Record where those observations came from. Without a baseline it is hard to assess what actually changed during the trial.
Choose a task owner and a person to review the result. Where possible, the review should not depend solely on the author of the instructions. You need an assessment of usefulness in ordinary work, not just confirmation that the system followed a prompt.
How do you prepare the material and a reference result?
Collect approved examples that represent several situations. Include valid data, missing information, contradictions and unusual material. Identify the sources and expected behaviour. When there is insufficient evidence, a correct result may be to identify the gap and stop before taking further action. If examples are generated, check the consistency of synthetic data and relationships between events (in Polish) before using them for evaluation.
Before the trial, prepare a reference answer or a list of features of a correct result. It need not be one perfect paragraph. A set of required information, unacceptable errors and sources to revisit is often useful.
Keep a version of the test set. If you change the documents and instructions at the same time, comparing results becomes harder. Distinguish material used to refine the solution from a later trial on new examples. As you make further changes, also check the independence of the AI evaluation set (in Polish), so repeating a known task does not replace an independent trial.
Which measures matter in the daily task?
Set criteria according to the consequences of an error. For a summary, these may include complete decisions, correct names and numbers, and the ability to return to the source. For classifying an enquiry, the right category and handling of an ambiguous case matter.
Measure checking and correction time as well. A quickly produced draft may transfer work to the next person. Record input preparation, tool operation, verification and corrections separately. Add the service cost using actual figures from the plan in use.
NIST's Measure function in the AI Risk Management Framework describes assessing and documenting system performance in the context of its use. It is a useful reference for a measurement plan. Your trial criteria still need to fit the task and the consequences in your company.
How do you separate output quality from operational safety?
A tool can produce good text while having overly broad access. Record both parts of the assessment. For integrations, check which sources the system can read and what actions it can take. Use the access scope approved for the trial.
State clearly whether the result stays a draft or may be sent or published. The guide to agent boundaries (in Polish) discusses approval rules. The guide to keeping a data-flow register (in Polish) can help you trace the material's path in your own register.
Prepare a case in which the tool lacks access or a service does not respond. Check whether the system shows an error and lets the user return to manual work. The trial should cover this behaviour too.
How do you write a pilot record?
Use a document that can be read before the start without a separate presentation. This is an example of a working structure, not a claim about results a company has obtained.
- Question: which decision the trial should support.
- Scope: task, tool, instruction version and test group.
- Material: origin, permission to use it and reference set.
- Assessment: criteria, calculation method and reviewer.
- Boundaries: permissions, cost, time and stopping conditions.
- Result: observations, errors, limitations and the next decision.

Also record what is outside the assessment. Testing one process does not establish that the tool is useful for every task in the company. Extending the conclusion requires new evidence.
How do you distinguish an error from an acceptable variation?
Before evaluating, record required elements and the room for variation. A summary may order sentences differently but should preserve decisions, names and sources. Adding a date that was never stated calls for a different response from awkward phrasing. Reviewers need a shared definition of these differences.
When assessments differ, keep the example and the reason for the decision. You can clarify a criterion, but the change should be visible in the set's version. Do not change rules merely to obtain the expected pilot result. If a new criterion matters, reassess earlier examples against it or clearly separate the stages of the trial.
How do you compare AI-assisted work with the existing process?
Prepare the same task scope for both ways of working. If a person previously received an organised document but the tool must first find the materials, the time difference will also reflect input preparation. State what you include and show the stages separately. Describe the measurement method alongside the data. Do not present your own calculation as a NIST requirement.
An exercise: compare preparation of a list of decisions from approved notes. The reviewer knows the required information but not which version used AI. This may reduce the effect of expectations on the assessment. It does not automatically make the assessment objective: you still need a way to resolve differing views of correctness.
Record cases where the tool refuses or needs help as well. Measuring usefulness only on successful runs leaves out the work of the person handling exceptions. Preparing for an AI audit (in Polish) describes gathering material and identifying its owners.
How do you calculate time and costs without an apparent saving?
A working balance can include preparing documents, writing instructions, running the tool, waiting, checking, correcting and handing over the result. Record waiting time separately from active human work. The team can then see whether a shorter process actually frees up time or merely shifts where it is spent.
Add the cost of the plan, additional services and support using current agreements or bills. For a shared licence, explain how a portion of the cost is assigned to the trial. Do not multiply the best single result across a whole year. First check task frequency, variation in material and maintenance effort.
An example criterion could be: further use requires useful results at an accepted review cost and no errors of a specified class. The process owner sets the actual numbers before testing. You can collect purchasing information in your AI tool inventory. The inventory guide (in Polish) can help you organise it, but the register does not replace measuring the work.
How do you define an error that stops the pilot?
Choose events by their consequences, not just language quality. A working set might include sending a result without required approval, adding a source that does not exist or reading material outside the permitted scope. Awkward wording calls for a different response from loss of control over data. Record the operator's expected response for each class.
A stopping condition should identify the decision-maker and how to preserve the information needed to investigate. One failure alone does not establish that the tool is useless for every task. It may justify narrowing the process or changing the integration. Do not resume until the cause is described and it is clear what will be tested again.
How do you prepare the final decision and the next trial?
Compare each condition with the observation and evidence. Identify differences between examples and cases requiring help. Instead of a single label such as “success”, record the task, material and access scope for which the result supports moving forward. Keep that separate from expectations about other uses.
Keep a decision on limited use and its conditions for change in your project's decision log. The guide to decision logs (in Polish) can help you structure the record. Add new examples and important earlier errors to the next trial. The team should be able to show both improvement and whether a new version has broken behaviour that previously worked.
A positive pilot result does not end quality assessment. After a change to the model, instructions, sources or tools, repeat the same task set and compare results against the preserved criteria. The guide to AI response regression tests (in Polish) describes keeping a reference set, classifying differences and recording the decision to admit a new version to work.
Criteria are easier to refine for a specific task, such as classifying messages with AI (in Polish). Compare error types and their consequences. A business experiment hypothesis card (in Polish), written before seeing the results, can help too.
What happens after the trial?
Repeat appropriate checks after a change to the model, integration or type of data. If the trial is intended to support a purchase, connect it with a Copilot renewal review (in Polish) or assessment of another specific order. You can discuss the pilot and review scope in the Aeneia Audit offer.











