Skip to content
Interface language: English
Audit / Signal

One AI test, two verdicts: how aggregation changes the result

Micro, macro and weighted F1 can favour different AI variants. Rework one fictional test and connect the aggregation rule to the decision it should support.

Article language: English

A person beside three stone-like discs mounted on a supported axle.
In this article

Two reports describe the same test. The first favours variant A; the second favours B. Neither has lost a row or confused the models. What differs is how they combine the results into one number. In a dispute like this, the calculator need not be broken. It may simply be answering a different question from the person making the decision.

Here, aggregation means summarising many assessments according to a chosen rule. Micro pools the outcomes across the whole dataset, macro gives each class an equal share in the summary, and a weighted average takes class sizes into account. Before choosing a single metric for a board report, check what its construction rewards in your task.

Consider a fictional test of an assistant that assigns questions to three categories: product card, delivery terms and access to materials. Each question has exactly one correct class, and each variant returns one label. Both receive the same hundred questions: eighty about the product card, ten about delivery and ten about access. There are no refusals to answer and no omitted cases. This is a numerical example only, not a measurement of any product.

What stays fixed in the comparison?

Keep the inputs, reference labels and predictions from both variants fixed. The predictions are the categories they assign. We do not change the model between calculations or select a more convenient sample. We will summarise the same answers in different ways. This lets us attribute a change in the recommendation to the aggregation method, instead of guessing what happened during another run.

Variant A correctly identifies all eighty product-card questions. It recognises four of the ten delivery questions and sends the other six to the product-card category. It does the same with access: four correct, six assigned to the product card. That gives eighty-eight correct answers in total. The product-card category attracts twelve questions that belong elsewhere.

Variant B identifies sixty-eight product-card questions, sends six to delivery and six to access. It recognises nine out of ten questions in each smaller class, incorrectly sending one to the product card. That gives eighty-six correct answers in total. Both variants make mistakes within the same three-category task, but distribute them differently.

The basics of evaluating message classification include defining labels and distinguishing types of error. Here, those definitions are already settled. We are interested in the next decision: what should become the single number used to compare two completed sets of predictions, and what that number will leave out. For more context, see the article on testing AI message classification (in Polish).

Before calculating, make sure a class name still refers to the same activity. If delivery and access were merged between exports, matching the column names is not enough. The task has changed. Do not compare the old classification with the new one as though only the averaging method were different.

Check which questions are included in both summaries. In this example, the list is identical. If one variant fails to return a label, you cannot simply remove that item from its report alone and still call it the same test. Agree in advance how to assess missing answers and show how many there were. A report may evaluate only the answers actually given, but it must then distinguish that assessment from the result for all assigned tasks.

When does micro favour the majority?

According to the precision_recall_fscore_support documentation, micro calculates metrics after pooling true positives and the two types of error across the full scope. Macro takes the unweighted mean of the class-level scores, while weighted averaging weights those scores by the number of examples that actually belong to each class.

In our task, each question receives one of three labels. Every wrong prediction therefore both misses the true class and incorrectly assigns the question to the predicted class. For A, the totals are 88 true positives, 12 false positives and 12 false negatives. Micro F1 is therefore 176 divided by 200, or 0.88. For B, it is 172 divided by 200, or 0.86.

F1 combines precision and recall into one measure. Here we use an equivalent formula: twice the true positives divided by the sum of twice the true positives, false positives and false negatives. We give the formula so that the calculation can be reproduced. It does not establish the cost of an error or an acceptable level of risk for a particular company.

For the complete task as defined here, micro F1 equals accuracy, the proportion of correct answers. The classification_report description explains why micro is not shown separately for a standard multiclass task covering all classes: it repeats the same information. That can differ when the report covers only a subset of classes or when multiple labels can apply at once.

A's advantage is therefore two correct answers out of a hundred. That is a real advantage on this metric for these data. It does not mean that A handles every type of question better. The eighty frequent cases have a strong influence on the total; failures in the two small categories take up less of it.

This result may matter for a particular objective, but that objective needs to be stated. If you are asking what proportion of cases in this dataset are assigned correctly, micro F1 answers directly. For this task, multiplying that value by the number of cases gives the number of correct assignments. If you want to assess all three service areas as equally important functions, you also need the class-level summary and the individual classes.

What changes when classes have equal weight?

Let us calculate F1 for each category. In A, the product card has 80 true positives, 12 false positives and no false negatives. Its F1 is 160/172, approximately 0.9302. Delivery has four true positives, no false positives and six false negatives: 8/14, approximately 0.5714. Access gives the same result. Macro F1 is the mean of those three values: approximately 0.6910.

Aeneia conceptual photography

For B, the product card has 68 true positives, two false positives and twelve false negatives, giving 136/150, approximately 0.9067. Delivery has nine true positives, six false positives and one false negative: 18/25, or 0.72. Access again gives the same result. Macro F1 is approximately 0.7822. By this method, B wins.

Not a single prediction has changed. What changed was the answer to the question about how to summarise them. Giving every class an equal share made the improvement in both small categories count for more. You must not present this difference as an extra quality gain achieved without changing the model. It is a different description of results you already had.

Macro F1 does not mean that a random question will be handled correctly with that probability. It is an average of class-level measures. Nor is it F1 calculated from average precision and average recall. The order of operations matters: calculate F1 for each class first, then average those values.

Equal weight does not automatically settle importance. A rare class may have serious consequences when misclassified, or it may be a relatively minor addition. A business weight needs its own justification. Choosing the name macro does not supply knowledge about the consequences of assigning a particular case incorrectly.

For a small class, look at the raw counts too. One example assessed differently in a group of ten can noticeably change the result. In your AI pilot evaluation criteria, record what evidence is sufficient for the decision in question and what is still missing. Our simple calculation does not establish a minimum sample size for every study. The related topic is explained in the article on criteria for evaluating an AI pilot (in Polish).

Will a weighted average settle the dispute?

Now use weights of 80, 10 and 10: the class sizes according to the reference labels. For A, we get approximately 0.8585. For B, approximately 0.8693. Weighted F1 also points to B, even though most questions concern the product card. It is not just another name for accuracy. It averages the already calculated class F1 scores, rather than the raw number of correct answers.

The calculation can be reproduced: for A, add 0.9302326 × 80, 0.5714286 × 10 and 0.5714286 × 10, then divide the sum by 100. For B, substitute 0.9066667 and two values of 0.72. We kept full values in the calculation and rounded only for display. Recalculating from rounded figures can produce a slightly different final digit.

Google discusses class-imbalanced datasets, distinguishing majority and minority classes and warning against relying on accuracy alone under severe imbalance. This explains methodology; it is not evidence that the proportions in our example describe any company.

State where the weights came from. Class sizes in a deliberately selected sample need not match the future flow of questions. If a team intentionally collected more difficult cases, the weighted average reflects that dataset. Without further assumptions, do not turn it into a forecast for the whole service.

A change in proportions over time is a separate issue, discussed under data drift and new questions. The dataset composition stays fixed in today's comparison. You can later examine a different future mix as a separate scenario, keeping a clear distinction between a measurement and an assumed simulation. This is discussed further in the article on data drift and changing questions (in Polish).

How should you use different averages in a recommendation?

Include all three summaries and the class counts with the decision. In our example, A has micro F1 of 0.88, macro of approximately 0.6910 and weighted of approximately 0.8585. B has 0.86, approximately 0.7822 and approximately 0.8693 respectively. The person reviewing the report should see why one comparison favours A and the other two favour B. We do not choose the winner by a vote among metrics: two votes do not automatically provide a better basis for a decision than one.

Then state the objective and limitations. You might choose B because it performs more evenly across the three functions, while explicitly accepting more product-card errors. Or you might retain A for product-card questions only and route the other cases differently. That choice needs a separate assessment of the routing process, however, because identifying the type of case was precisely where A made mistakes in this test.

Thumbs-up and thumbs-down ratings from users cannot replace those reference labels. They describe reactions from people who chose to leave them. Similarly, an independent human assessment before the AI suggestion requires the measurement conditions to be set first. Each of these information sources has its own unit of assessment and limitations. For the related method, consult the article on interpreting user thumbs-up and thumbs-down ratings (in Polish). Further guidance is available in the article on independent human assessment before an AI suggestion (in Polish).

For a conversation with Aeneia Audit, bring non-confidential predictions, reference labels and a description of the decision the report should support. Retain the dataset version and the rules for omitting records. This makes it possible to check whether the average answers the company's question, instead of choosing whichever makes the preferred model look best.

The report's closing statement should identify the chosen variant, the criterion, the weaknesses consciously accepted and the evidence still missing. A useful acceptance check is to have a second person independently reproduce one class score and one aggregate. Give them the input counts, weighting rule and rounding method. If their result differs, identify where the calculations diverge before changing the recommendation. The cause could be a different set of classes, an omitted item or a different order of operations. A display's own rounding must not quietly become a new comparison method.

A list of three averages alone would leave the reader with the same task we started with. The numbers should help a decision, rather than quietly hand it over to the spreadsheet's behaviour.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear