Fast demo, slow queue: testing AI under load
Test AI under load by separating queue time, first content and the complete result. Compare timing distributions, errors and completeness in a controlled trial.
Article language: English

In this article
During the demonstration, the answer starts almost immediately. A few people open the tool at once, and suddenly the first paragraph takes time to arrive. Text then starts arriving quickly, but the complete list is still not ready. “AI responds in two seconds” does not specify whether that means the first text or the complete result.
Under load, you need to see the wait before a request is sent, the time to the first content and the time to receive a complete result separately. You also need the distribution of measurements: a good average will not tell you how long the slowest cases wait. The test should concern a specific task and an agreed environment, without springing an unannounced stress test on a working company.
Consider a fictional assistant that summarises equipment records. An employee submits a document and receives a list of five agreed pieces of information. We want to check one person's work and several people sending similar tasks at the same time. We are not reporting a provider benchmark here. We are preparing a test scope that a contractor can run in an authorised environment.
When do you start the clock?
Record the path of a single task. The employee selects material, the application accepts the job, perhaps waits until it can send it, and then submits the request to the service. Only afterwards do the first content and the rest of the result return. If the clock starts at the external API call, it excludes the earlier wait in the local queue.
The GenAI-Perf metrics documentation distinguishes the time from sending a request to the first response from the time to the final response. It also shows distributions and throughput. NVIDIA marks the tool as being phased out and points to AIPerf for new needs; we use the measurement definitions here, rather than recommend an installation.
For our acceptance review, name four moments: the application accepts the task, the request is sent to the service, the first useful content is received, and the complete response is received. Define each so the person measuring knows what to record. A first technical response containing no content is not necessarily the moment when the user has something to read.
Your own measurement of the interface may also include the content appearing on screen. The backend receiving the first fragment and the interface displaying it are different observations. If you are assessing the employee's experience, define that scope too. Do not compare one report's service time with another's whole-application time and attribute the difference to the model alone.
Microsoft's discussion of Azure OpenAI performance and latency identifies the model, input length, output length and load as factors affecting time. It recommends pairing timing observations with token counts. A token is a unit of text used by the model; it does not necessarily correspond to one word.
For each task, therefore, keep the input and output sizes in the unit appropriate to the measurement, the available configuration and the completion status. In our example, the five-field list should have a comparable scope. If a changed instruction makes the assistant write a multi-page essay instead, the longer time does not yet explain whether performance on the same task has changed.
The article on accepting a streamed AI response (in Polish) develops the boundaries of completeness. The first text may appear quickly and remain incomplete after an interruption. Do not put that run in the “result ready” column based on the first sentence appearing. Time and status must refer to the same stage.
How can you increase load while keeping comparisons fair?
Choose a set of non-confidential documents with a defined scope. Include varied cases reflecting real work, but use the same material when comparing load levels. Sending one short sentence for a single person and long files for a larger group tests two changes at once.
Distinguish the number of tasks running simultaneously from the arrival rate of new jobs. In the first mode, the test maintains a specified number of active requests: a new one starts when a previous one finishes. In the second, new tasks arrive at an agreed pace regardless of whether previous ones have finished. These represent different working conditions.
For the assistant, gradual levels might be one, three and six concurrent cases. These are working values for the example, rather than a recommended capacity for a particular account. The person responsible for the environment agrees the permitted scope, costs, duration and stopping condition. The test should not change data or block ordinary service.
Record traffic from other processes using the same service or limit too. An overnight import may consume shared throughput even though only a few employees appear on the assistant's screen. If it cannot be separated, treat it as a known test condition. Without that information, later comparisons are difficult.
Label warm-up and first-run conditions separately. Some tasks may use resources prepared earlier or repeated material. Do not remove slow initial runs because they spoil the chart, or present a series of identical inputs as a complete picture of everyday work. Keep a description of what was deliberately repeated.
Also check whether the tool sending test tasks actually maintains the intended pace. If it cannot generate requests fast enough, the server receives less load than planned. The report should show actual send times, rather than only the configured value.

For a larger batch, maintain the mapping between your responses and records. A separate guide to batch results and records (in Polish) discusses this. A late completion must still belong to the correct input and run. A fast measurement with responses swapped confirms nothing useful. Quality requirements remain in force when checking time.
Include a return to lower load after the higher-load series. If the first single-person run was fast but a later single task still waits in a backlog, the report should show that state. Do not combine it without explanation with the quiet start. The sequence of test series and the time allowed to account for earlier tasks form part of the measurement conditions. Otherwise, two apparently identical concurrency levels may represent different states of the overall process.
Do not increase document length, tool count and load together if you want to establish the cause of a difference. You can later test a realistic combination of those conditions as a separate scenario. Changing one thing at the start makes it easier to understand where the queue forms and what actually limits delivery.
How do you read timings without losing the slowest cases?
Keep individual measurements as well as summaries. The median shows the middle of an ordered set; an appropriately defined high percentile describes its slower part. State the number of runs and the calculation method. With just a few observations, an apparently precise percentile does not establish stable behaviour in a large service.
A simple invented calculation shows the limit of an average. Seven tasks take two seconds each; one takes eighteen. The average is four seconds. Nobody received a result in exactly four seconds, and one person waited nine times as long as the others. Keep the slow result in the report so its cause can be investigated.
Do not combine different stages in one distribution. Show the local queue, time to first content and time to the complete response separately. Generation after it starts may remain similar while the wait before sending grows. That calls for a different fix from a slowdown in the arrival of subsequent fragments.
For an unsuccessful result, keep its status and the time until the failure was recognised. Do not count a quick refusal or rate-limit error as an efficiently completed task. If successful-run timings exclude failures, say so explicitly and give the failure count. Otherwise, an overloaded solution may appear faster because it immediately stops answering many requests.
The NVIDIA documentation on goodput describes completed requests per second that satisfy specified metric constraints. This differs from overall throughput. The thresholds in the documentation's examples are illustrative; the company must define its own conditions for usefulness.
For our task, such a condition might require receiving the complete list within the agreed time. We still check separately whether the five fields contain correct information. A timing condition does not establish substantive quality. The report can therefore show on-time completions, late completions and results rejected after review.
For an API 429 response, indicating a request-rate limit, turn to the shared API budget and queue behaviour (in Polish). In this review, record the response and subsequent wait without automatically multiplying attempts to improve the result. Performance testing and the design of rate-limit handling are related, but should not conceal each other's costs.
Do not compare token rates alone if the usefulness of the response has changed. A shorter but incomplete text may be generated faster. The task owner needs the time to the correct result. Measuring work with AI (in Polish) then adds review and correction time; neither is included in the service measurement alone.
What should you do with the test result?
If the queue grows at the selected arrival rate, describe that scope rather than announcing one capacity figure for the entire AI system. The result applies to the material, configuration and conditions actually checked. Changing response length, model or shared traffic may change the observation. Keep the data needed to reproduce the comparison.
Prepare an operational decision: limit concurrency, move less urgent batches, change the response scope or route particular tasks elsewhere. Each proposal needs its own verification. Do not infer from a single overload that replacing the model is the only solution, or that extra retries will clear the queue by themselves.
Check what happens after new arrivals stop. How long does the backlog remain? Do new, short cases wait behind long ones? Can users see that their task is still accepted? The article on work continuity after an AI failure (in Polish) can help plan a response. Excessive waiting may also require a deliberate handover of work.
Include a job the user cancels while waiting in the acceptance review. Establish whether only the view disappears or further execution actually stops. Work still waiting in the background may continue consuming available resources and affect the remaining measurements. Keep its status in the job count. Removing it from the report alone does not remove it from the process. Account for these tasks after the test too, so the next run does not begin with an undocumented backlog.
For a discussion with Aeneia Audit, prepare a description of one task, typical material and information about other processes using the same service. This is enough to define a question worth measuring.
Finish the report with results for each condition, the number of failures, measurement limitations and the next action chosen. The demonstration's two seconds may remain an accurate observation of one run. The company also needs results for the agreed concurrent workload.







