Assessing an AI vendor before you buy: what evidence to ask for
Assess an AI vendor using evidence about data, access, service terms, task quality, support and exit options. A practical guide to a scoped buying decision.
Article language: English

In this article
The demo is impressive: the tool makes light work of a carefully prepared task. Before buying, you still need to check how it handles your own data, errors, service terms and the end of the relationship. A vendor assessment should lead to a decision about a specific use case and the scope of a trial. A polished interface can get a conversation started. The purchase order needs a firmer basis.
The outcome might be a purchase with a defined scope, a limited trial, a request for more evidence or a decision to walk away. You do not need a ranking with a deceptively precise score. If the vendor cannot confirm a requirement you consider essential, a high score for interface design does not resolve the problem. The approach below helps turn interest in a product into a reasoned decision.
Start with the process owner's requirements
Describe the task, how it is done today, who receives the output and what happens if it is wrong. Add the types of material involved, the integrations you need and whether the output will remain a draft. Your questions will then concern actual work. A broad question such as “Is your AI secure?” usually gets an equally broad answer.
Separate essential requirements from useful extras. The ability to restrict access might be a condition of purchase; a convenient editor might simply be an advantage. Make that distinction before the presentation. If you have several ideas, select the task from your own use case register, so the feature you buy has someone responsible for it and a clear purpose. A guide to preparing an AI use case register (in Polish) can help you make that register.
Establish exactly what the vendor is selling
Identify the contracting party, the plan, how the service is delivered and the components it uses. A product may combine the vendor's interface with models from other companies, search and additional services. Ask which components the documents cover. The terms for a well-known underlying model may not describe the whole product you are buying. If the offer promises local operation, check what actually works offline and which dependencies remain (in Polish).
Ask for dated documents that apply to the proposed plan. Distinguish an available feature from an announced one, and a contractual term from a sales statement. If the vendor offers a custom configuration, establish how it will be confirmed and who will maintain it. Keep the link, document version and the name of the person who confirmed the interpretation of a particular term in your assessment record.
Follow the data through one task
Show the vendor an example of the material's route: uploading a file, processing it, generating an answer, saving the history and exporting the result. Ask where copies are created and who can use them. Check separately whether data is used to improve models, stored to support a feature or kept in security logs. These are different questions that need separate answers.
For example, the OpenAI API data controls documentation distinguishes use for training, log retention and application state. Do not automatically apply that document to a different product or an intermediary's service. Prepare your own map of storage locations and check the deletion conditions that apply to your service. The guides to mapping AI data flows (in Polish) and planning retention and deletion (in Polish) can help structure that work.
Check access controls before connecting resources
Ask to see the administrative roles, how users are invited and removed, and how integrations receive access. Establish whether the tool can be restricted to the resource needed for the task. The label “email integration” does not tell you whether a product reads a selected folder, the entire account or data available to the organisation.
Use an environment and material approved for the assessment. Test that the tool can read an approved resource and is denied access to a resource outside the agreed scope. Record what appears in the event log and how to disconnect the integration. The review of AI integration permissions (in Polish) explains the practical difference between user consent, application permissions and what the application can do. The consent screen alone does not complete this assessment.
Match each piece of evidence to a requirement
Give each requirement a status: confirmed by a document, demonstrated in a trial, stated without evidence or not met. That record is more useful than a single average across all the answers. It also shows what additional information could change the decision. Restricted documents can be inspected under an agreed arrangement, with a clear record of what you were allowed to see.
Check which entity, period, product and scope a certificate, assessment report or security description covers. Do not assume it applies to every add-on and configuration. If a document leaves something unclear, record a question for confirmation. Preparing materials for an AI audit (in Polish) describes a useful index of evidence, responsible people and access restrictions.
Test against your own quality criteria
Agree representative tasks and expected behaviour before looking at the results. Include missing information, conflicting data, denied access and the need to return to the source. The vendor can help configure the solution, but the assessment should cover the team's normal way of working. A vendor-prepared demonstration and a task you carry out yourself provide different kinds of evidence. If the output feeds another application, include structurally valid JSON with incorrect meaning (in Polish) in the trial, so format validation does not replace a check of the data.
Record the time spent checking and correcting the answer as well as generating it. Establish whether the trial is paid, which settings will be retained and when access ends. The AI pilot evaluation criteria (in Polish) provide an acceptance plan. Evidence that a tool is useful for one task should not be extended to other processes without further assessment.

Ask about changes to the model, features and terms
The product will change. Establish whether the vendor notifies you of significant changes, which settings can be kept fixed and whether a new version can be tested. If you cannot identify the model or service version in use, record that as a limit on reproducibility. The company can still make a decision, but it should understand the basis for it.
Prepare your own set of cases to reassess the task after an update. Decide who monitors notices, runs the test and approves continued use. AI response regression tests (in Polish) describe this kind of comparison after a change. This part of the assessment helps prevent an old pilot result from becoming the only evidence for years to come.
Test the support arrangements against a specific failure
Ask how to report lost access, incorrect behaviour and a suspected data incident. Check the channels, operating hours, escalation procedure and the information the customer will receive. If the offer includes availability or response commitments, read their definitions and exclusions. Waiting for a salesperson to reply is not an agreed support procedure.
Consider a hypothetical example: the product stops retrieving documents but continues to answer using older material. You need a way to detect the problem, limit use and inform the people receiving the output. The vendor should explain which signals you will receive. The customer's team, in turn, needs to know who decides when to switch to doing the task manually and when to restart the service.
Work out the cost of your intended use and the route out
Ask for a pricing model covering user access, consumption, additional features and support. Check the minimum commitment, renewal arrangements and changes to scope during the contract. Calculate the cost for your process using assumptions you can justify. If you do not yet know the likely volume of use, show scenarios and identify the data needed for a more precise estimate.
At the same time, test exporting your materials, instructions and outputs, and disconnecting integrations. Establish what happens to the history and who receives confirmation that the service has ended. The cost of leaving belongs in the purchase decision. We discuss this in the total cost of an AI process (in Polish), where the licence is only one part of the bill. An AI vendor exit planning guide (in Polish) will help you prepare a trial transfer of your own materials and process.
How to keep a useful offer comparison
Use the same questions and the same task for every solution under consideration. Create columns for the requirement, answer, evidence, limitation and effect on the decision. If one offer includes implementation and another only access to the product, show that difference separately. Comparing prices without the scope would suggest a similarity between the services that you have not established.
When an answer is missing, distinguish three situations: the vendor may not have the feature, may not provide evidence or may be waiting for clarification from a subcontractor. Each calls for a different action. Set a deadline for the missing information and name the person who will check the reply. If the gap concerns an essential requirement, the purchase remains undecided. Do not substitute zero or a favourable assumption for an unknown value just to make the table look complete.
Involve the people responsible for the process, data, configuration and financial commitment. They do not all need to attend every presentation. They should, however, confirm their part of the requirements before the decision. Agree how to resolve a conflict when user convenience needs broader access than the resource owner allows. That conflict tells you something about the limits of the purchase. It should not be hidden in an average score.
A worked example: an assistant that searches documents
A company is considering a service that answers questions using internal instructions. It must respect document access permissions and identify the source material. The demonstration produces an answer that looks correct. In its own trial, however, the team also checks an account without access to a selected directory, an outdated instruction and a question the documentation does not answer. It assesses the behaviour in each case separately.
If the solution summarises well but cannot show which version of a document it used, the process owner might restrict the trial to a less demanding task or pause the purchase until the issue is clarified. This is an example of how to reason through a decision, not an account of a real deployment. It connects a requirement, an observation and a decision. That record gives you something to return to when the product's features or offer change.
Record the decision and its conditions
The decision record should name the use case, product, data scope, evidence, gaps and accepted limitations. Assign responsibility for the purchase and for later reassessment. If approval is conditional, state what must happen before real materials can be used. “Approved with comments” leaves too much room for interpretation when nobody knows which comments must be addressed before work starts.
- Essential requirements have evidence or an explicitly recorded gap.
- The trial uses a real task and takes into account who will receive the result.
- The data and integration scope matches the approved use.
- The cost includes the team's work and the end of the relationship.
- There is an agreed way to assess changes and failures.
After buying, turn the findings into rules for using the approved service, so employees understand its limits. The guide to writing an AI-use policy (in Polish) can help organise those rules. If you need to organise the assessment before making a commitment, see the Aeneia Audit service.
For a product that retrieves knowledge, ask the vendor to demonstrate source and answer quality using your prepared material. The guide to checking source and answer quality (in Polish) explains what to examine. Prepare a B2B vendor selection brief to organise the offers; the guide to preparing a vendor selection brief (in Polish) explains what to include. In both cases, what matters is whether you can test the claimed feature using your own, properly prepared material.











