Skip to content
Interface language: English
Audit / Signal

AI forecasting service demand: test what was knowable at the time

Evaluate service-demand forecasts using information available at the decision time, a simple baseline and errors at the right horizon.

Article language: English

A person lifts a sheet from an open sample case on a wooden table beside a simple meal.
In this article

The best forecast for next week was produced on Friday, after the week had ended. It received the final numbers, corrected enquiry categories and the campaign outcome. It explained almost everything perfectly. Unfortunately, the team roster had to be prepared the previous Thursday, when none of that information was available.

This fictional example shows the difference between reconstructing the past and testing a forecast. A past week can be used for evaluation if the model uses information available when the forecast was issued. Later information answers a different question from the one faced by the person planning service coverage.

Start acceptance with the forecast issue date. Establish what the team knew then, the period for which it predicted case volumes and the decision it had to make. Only then choose evaluation material, a simple baseline and a way to describe error.

Thursday has its own state of knowledge

Suppose a workshop plans coverage for enquiries every Thursday morning for the following week. The model must predict new cases for each working day. This is different from predicting tomorrow's result at the end of a day or a total for the following month. The lead time, called the forecast horizon, is part of the task.

Record the cutoff and its time zone. What is in the data at eight on Thursday? The previous day may already be closed, or some channels may still be awaiting import. The week's figure in today's report need not match what was visible then. A test needs the state available at forecast time, or an honest account of the limits of reconstructing it.

For each additional input, identify when it becomes known. A campaign plan for next week may be available beforehand. Its actual number of clicks is not.

If the future plan is not yet approved, separate scenarios. A forecast with the campaign and one without it answer questions under different assumptions. Label them as scenarios, not contradictory certainties. Record who will confirm the choice before the result is used in the roster. After the period ends, check which scenario actually occurred; otherwise you will compare reality with a forecast for conditions that never happened. Planned future information is different from a result learned later, even when both fields have similar names. Likewise distinguish planned opening hours from the eventual handling time for future enquiries. The week’s date does not make both available in advance.

Watch for later corrections. A case initially marked as a product question may be reclassified as a complaint after a conversation. Today’s archive shows the final label. Inserting it into a historical forecast as though it were known immediately gives the model information learned later. A case creation date does not establish the availability of every field.

The same problem arises during data cleaning. If a method of filling gaps or transforming values was determined using the whole period, check whether it also used the future evaluation portion. The person accepting the result need not know every formula. They do need a description of which decisions were made before the forecast date and which data informed them.

This is a particular case of AI test-set leakage (in Polish). Here the boundary also depends on when knowledge was available. Chronological rows help, but do not guarantee that later information has not been added to an earlier row.

Test successive issue dates under the same conditions

Instead of checking one convenient week, choose successive historical issue dates. For each, reconstruct the permitted inputs, prepare the same kind of forecast and compare it with the outcome learned later. This is a rolling test: the point from which you look ahead moves through history.

The scikit-learn TimeSeriesSplit documentation describes splits that train on earlier observations and evaluate on later ones. Its gap parameter can omit a specified number of observations before the test portion. The documentation states that observations should be equally spaced for comparable metrics across test portions. The split itself does not control the availability of every data field.

In our planning example, the gap should reflect the actual situation. If Wednesday's data is not yet complete on Thursday, do not pretend it was available in the test. Equally, do not add an arbitrary gap just to make the report look more cautious. Agree what the team can really receive on time and how long input preparation takes.

Keep the horizon the company will use. A good forecast for tomorrow does not establish the quality of a forecast for the whole following week. Examine results separately by distance from the issue date if the decision requires it. A person preparing the roster should see where the result ceases to be useful enough, rather than receive a single average across all days.

Choose different conditions too: an ordinary rhythm, a period of greater interest, days after service closure and a change in incoming channels. Do not treat one successful test as representative of a whole year. If history does not cover an important event, record that gap. A model checked during a quiet month does not thereby receive a certificate for the holiday peak.

Do not adjust settings after seeing each result and then count all those weeks as an independent final evaluation. Separate the portion used to refine the solution from a later test of a fixed version. Record when the model, data preparation rule or error measure changed. Otherwise one table combines different solutions under a common name. For the same saved input version, also compare successive runs of the AI tool (in Polish) if the process itself may vary between runs. This is a different kind of instability from data changing on successive days.

Aeneia conceptual photography

Keep a small evidence pack for each issue date: input identifier, solution version, preparation time and the forecast itself. Add the observation used for evaluation and the error calculation only later. If the old input can only be approximated, describe which fields or corrections could not be checked. Do not call such a test a faithful reconstruction of daily work. For subsequent real forecasts, save the state used in the decision at the time; later evaluation will not depend on rebuilding history from today's database.

For an incomplete period, first establish what was actually observed. The distinction between a gap and zero is discussed in whether an AI alarm describes an event or missing measurement (in Polish). In forecasting, the same gap can corrupt both the input and the later reference against which the result is compared.

A baseline need not have an impressive name

Compare the model with a simple rule the team could use without it. For a given process, that might mean repeating an appropriate earlier period. Choose and describe the rule before comparing; it should not receive worse data or less preparation time merely to make AI win.

Forecasting: Principles and Practice presents a naive method that repeats the last observation and a seasonal naive method that uses the latest observation from the corresponding season. They are simple baselines for evaluating other methods.

In our workshop, a rule using the latest completed matching weekday whose result was available when the forecast was issued may be worth testing if it fits the data's rhythm. For a forecast issued on Thursday at 8:00, do not use the unfinished result for that Thursday or the later Friday. If a result is delayed or missing, follow a procedure agreed in advance or exclude that forecast from the comparison; apply the same availability cutoff in the historical test. If a more complex model does not help with the agreed decision, the label "AI" alone does not justify additional work. Conversely, one better result for a simple rule does not settle the whole comparison. Examine the distribution of errors across periods.

Do not hide error direction. Underestimating cases may leave the team with a backlog, while overestimating leads to a different organisational cost. Agree which mistakes matter more to the process and how to show them. The criterion should follow service planning, not the metric that makes the model look best.

A fictional numerical example: 50 cases arrived on each of two days. Forecasts of 10 and 90 give the correct total of 100 but miss each day by 40. Forecasts of 40 and 60 also total 100, with daily errors of 10 each. If the decision concerns staffing each day, agreement on the total alone is not enough to choose a solution.

In both cases positive and negative errors can cancel in the sum. Therefore show the size of errors regardless of sign alongside the total difference. Here the average absolute daily error is respectively 40 and 10 cases. This does not establish that the second model is tested for the future: it shows only why one matching total conceals a different quality of daily plan. In a real test keep the daily results so the team can relate them to its decision.

Show results at the level at which someone will act. A weekly total may help plan overall availability; a daily distribution may help set shifts. Do not evaluate only one if the commissioned process needs both. During acceptance, select several specific errors and show how they would change the team's decision.

Make uncertainty useful for planning

If the solution returns an interval, establish what it describes. Do not treat two numbers beside a forecast as an acceptable range intuitively chosen by the author. You need to know the future value, period and stated level to which the result refers.

The same forecasting textbook describes a prediction interval as a range in which a future observation is expected at a specified probability level. It describes forecast uncertainty; it does not guarantee that every next value will lie inside it.

In the test, check how often and by how much values fell outside the bounds, and how wide the range was. An interval covering almost every imaginable number may not help set a roster. A narrow interval that regularly misses the outcome should not create an appearance of certainty either. Evaluate the agreed horizon and conditions, acknowledging the limited number of historical trials.

Do not directly turn a forecast number of cases into a required headcount. Cases can require different amounts of work, and the team has particular skills and availability. That next planning step is discussed in seasonal enquiry peaks and service capacity (in Polish). An accurate message count does not itself settle who should answer or how long each task will take.

If the type of incoming cases changes, an old result may need reassessment. A new form, product or contact method can change the relationship between volume and workload. The article on data drift and changing questions for AI (in Polish) develops this point. Mark such changes in forecast history rather than explaining every error as a generally "difficult week".

Agree what happens if the forecast itself is late. A result prepared on Friday does not replace Thursday's material if the roster is already approved. Record whether the team should then use its previous rule, postpone a specific decision or activate an agreed manual process. Number quality and delivery time are two conditions of usefulness for the same solution.

For a discussion with Aeneia Audit, prepare one issue time, the information available then, the recorded forecast and the later outcome. Keep the simple comparison rule and a description of the team's decision as well. This starts the conversation without pretending that a past Thursday had access to Friday's knowledge. A good report should be able to stop that journey through time.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear