1 Averaging across queries in information retrieval

1.1 Why query-level aggregation is needed

Many information retrieval (IR) evaluation measures are defined per query. A system’s output is assessed by comparing its ranked results with the ground truth for each query, producing a query-specific score. To compare systems globally, those query scores must be combined into a single number that summarizes performance across the whole test set. The aggregation rule determines how each query’s evidence contributes to the final result.

1.2 Macro vs micro: core intuition

Macro aggregation combines query-level metric values by treating each query as one unit of influence, regardless of how many relevant items the query has or how many ranking positions contain meaningful signals. Micro aggregation instead pools underlying counts (such as true positives and false positives, or other event totals) across queries and computes the metric from the pooled totals. As a result, queries with larger volumes of outcomes tend to dominate micro scores.

1.3 When averaging choices change conclusions

If all queries are similar in difficulty and have comparable numbers of relevant items, macro and micro often produce similar rankings of systems. Disagreement is most likely when the test set contains queries with very different relevance set sizes, when some queries are systematically harder, or when a system’s errors concentrate on particular subsets of queries. In those cases, the averaging scheme can change which system appears best.

2 Definitions and notation

2.1 Query-level metric computation

Let a test collection contain queries \(q \in Q\). For each query \(q\), a ranked list is produced by the system, and the metric \(m(q)\) is computed from that list and its relevance judgments. Some metrics can be expressed directly from counts (e.g., precision and recall), while others depend on the ordering (e.g., average precision and NDCG).

2.2 Macro aggregation (unweighted across queries)

Macro aggregation computes the mean of query-level scores: \[

M_{\text{macro}} = \frac{1}{Q}\sum_{q \in Q} m(q).

\] Each query contributes equally, so a query with sparse relevance has the same influence as one with dense relevance.

2.3 Micro aggregation (pooled counts across queries)

Micro aggregation pools event counts across queries before computing the metric. For count-based measures, one first accumulates numerator and denominator components over all queries. For example, when a metric depends on true positives (TP), false positives (FP), and false negatives (FN), micro precision and recall are computed from totals: \[ \text{Precision}_{\text{micro}} = \frac{\sum_q \text{TP}(q)}{\sum_q (\text{TP}(q)+\text{FP}(q))}, \quad \text{Recall}_{\text{micro}} = \frac{\sum_q \text{TP}(q)}{\sum_q (\text{TP}(q)+\text{FN}(q))}. \] This mechanism implicitly weights each query by the volume of outcomes it contributes.

2.4 Handling varying numbers of relevant items per query

Relevance judgments typically vary in cardinality across queries. Macro aggregation is unaffected by those differences because it operates on \(m(q)\) values directly. Micro aggregation changes with the distribution of relevant items because queries with more relevant documents (or more evaluated positions) contribute more counts to the pooled sums. Therefore, the effective weighting is determined by dataset structure rather than by an explicit averaging choice alone.

3 Macro averaging details

3.1 Equal weighting of queries

Because macro scores average \(m(q)\) across all queries, each query is treated as an equal unit of evaluation. This is often interpreted as prioritizing consistent performance: a system that improves a small number of difficult queries can receive a noticeable macro benefit, even if those queries have few relevant items.

3.2 Sensitivity to outliers and hard queries

Macro aggregation can be sensitive to queries whose metrics differ substantially from the rest. A handful of very low-scoring queries may pull down the overall macro average, particularly if those queries share a specific failure mode (for example, a mismatch in vocabulary or intent). Conversely, if a model dramatically improves performance on a small set of queries, macro may reflect that gain more strongly than micro.

3.3 Common macro metrics

3.3.1 Macro-averaged precision

When precision is computed at a fixed cutoff (such as precision@\(k\)), each query yields a precision value based on the top \(k\) retrieved documents. Macro-averaged precision is then the arithmetic mean across queries: \[

P@k_{\text{macro}} = \frac{1}{Q}\sum_q P@k(q).

\] If some queries return few relevant items, their low precision affects the final score proportionally, despite any differences in relevance set size.

3.3.2 Macro-averaged recall

Recall@\(k\) is computed per query using the number of relevant items retrieved within the top \(k\). Macro recall averages these per-query recalls: \[

R@k_{\text{macro}} = \frac{1}{Q}\sum_q R@k(q).

\] Queries with fewer relevant items can produce recall values that fluctuate more easily between high and low values; macro aggregation preserves that variability rather than smoothing it through pooled counts.

3.3.3 Macro-averaged F1

F1 at cutoff \(k\) can be computed from per-query precision and recall, then averaged: \[

F1@k_{\text{macro}} = \frac{1}{Q}\sum_q F1@k(q).

\] Because F1 is nonlinear in precision and recall, macro averaging can respond differently to trade-offs across queries than averaging a linear quantity would.

4 Micro averaging details

4.1 Pooling true/false positives across queries

Micro precision and recall rely on pooled counts. For a fixed cutoff such as \(k\), each query contributes up to \(k\) retrieved items to the event pool. True positives are summed across queries (relevant retrieved items), and false positives are summed as retrieved nonrelevant items. This produces a global precision that reflects how often retrieved items are relevant across the entire test set.

4.2 Weighting by query size or support

Micro aggregation inherently weights queries by how many relevant and nonrelevant items they contribute to the totals. In practice, this often corresponds to factors like:

  • the number of relevant documents per query (affecting false negatives and recall),
  • the number of retrieved positions being evaluated (affecting true/false positives),
  • the presence of judgment density (how much of the top-ranked region has labeled relevance).

As a consequence, a query with many relevant items or a large labeled footprint can outweigh a query with few labels.

4.3 Common micro metrics

4.3.1 Micro-averaged precision

For precision-based measures with count decompositions, the micro score uses pooled TP and FP totals: \[ P@k_{\text{micro}} = \frac{\sum_q \text{TP}@k(q)}{\sum_q (\text{TP}@k(q)+\text{FP}@k(q))}. \] This treats every retrieved item across all queries as an equally weighted event.

4.3.2 Micro-averaged recall

Micro recall pools true positives and false negatives across queries: \[ R@k_{\text{micro}} = \frac{\sum_q \text{TP}@k(q)}{\sum_q (\text{TP}@k(q)+\text{FN}@k(q))}. \] Because \(\text{FN}\) reflects relevant items not retrieved within the evaluated region, queries with more relevant documents have a larger impact on the denominator.

4.3.3 Micro-averaged F1

A micro F1 score can be computed by plugging micro precision and micro recall into the F1 formula, or by using pooled counts directly when a decomposition exists. The typical approach is: \[ F1_{\text{micro}} = \frac{2 \cdot P_{\text{micro}} \cdot R_{\text{micro}}}{P_{\text{micro}} + R_{\text{micro}}}. \] This differs from macro F1, since macro averages per-query nonlinear values while micro combines counts first and then applies nonlinearity once.

5 Relationship between macro and micro results

5.1 Interpreting divergence between the two

When macro and micro scores disagree, the system’s performance is uneven across the evaluation set. If macro is higher than micro, it suggests improvement in queries that are “small” under micro weighting (for example, queries with fewer relevant documents) or that lift per-query scores broadly. If micro is higher, the system likely improves many retrieval events on “large-support” queries even if some individual queries remain weak.

5.2 Influence of class imbalance across queries

Different queries can exhibit different class imbalance within their ranked results. For recall, queries with many relevant documents contribute more false-negative mass when the system misses them. For precision, every nonrelevant retrieved item counts against the pooled precision in micro mode. Therefore, imbalance in relevance density tends to amplify the gap between macro and micro interpretations.

5.3 Corner cases (e.g., queries with no relevant items)

Evaluation can encounter queries with no relevant items in the judgment set. Depending on the metric definition, precision-based measures may be well-defined while recall-based measures may be undefined (because the denominator is zero). Macro aggregation must decide how to treat such queries—either exclude them, impute a value, or handle them with a metric-specific convention. Micro aggregation can be less sensitive if pooled denominators remain nonzero, but it can still be affected depending on how events are counted.

5.4 Stability vs representativeness trade-offs

Macro scores emphasize representativeness across queries; they answer questions like “How well does the system perform on a typical query?” Micro scores answer “How relevant are retrieved items overall?” Macro can be more stable against extreme differences in query volume, but it can also be less stable to rare, atypical failures. Micro often produces lower variance when query sizes vary greatly, but it may obscure poor performance on particular query types by averaging them into a large pool.

6 Metric-specific considerations

6.1 Precision@k and recall@k aggregation

Precision@\(k\) and recall@\(k\) are commonly reported with either macro or micro aggregation. The choice interacts with the cutoff:

  • With precision@\(k\), micro treats every retrieved position across all queries as one event; macro treats each query’s top-\(k\) outcome as one unit.
  • With recall@\(k\), micro weighting strongly reflects how many relevant items exist per query, because the recall denominator depends on that support.

Therefore, selecting an averaging rule changes both the meaning and the sensitivity of these cutoff metrics.

6.2 Average Precision (AP) and mean Average Precision (MAP)

6.2.1 Macro-style averaging over queries for MAP

Average Precision (AP) is defined per query using the precision values at ranks where relevant items are retrieved, then averaged over queries to obtain mean Average Precision (MAP). This is naturally a macro-style scheme: \[

\text{MAP} = \frac{1}{Q}\sum_q \text{AP}(q).

\] Because AP is already a query-level summary of ranking quality, the usual MAP computation gives each query equal influence.

6.3 NDCG aggregation across queries

6.3.1 Averaging NDCG scores by query

Normalized Discounted Cumulative Gain (NDCG) is computed per query by comparing the system’s ranked list against an ideal ranking, then dividing by the ideal DCG. Standard practice aggregates NDCG by averaging the per-query NDCG values (macro-like): \[

\text{AvgNDCG} = \frac{1}{Q}\sum_q \text{NDCG}(q).

\] Since NDCG depends on positions and relevance ordering within a query, direct pooling across queries is generally not used in common implementations.

6.4 Mean Reciprocal Rank (MRR) aggregation

MRR uses the reciprocal of the rank at which the first relevant item appears for each query, then averages across queries: \[

\text{MRR} = \frac{1}{Q}\sum_q \frac{1}{\text{rank}_q}.

\] This averaging is also typically macro-style because it weights each query equally by its “first-hit” quality rather than by the number of relevant items.

7 Practical selection guidelines

7.1 Choosing macro when fairness across queries matters

Macro averaging is a good default when test queries are meant to represent different user intents or scenarios equally, and when it is important not to let a handful of high-support queries dominate the overall outcome. It is also helpful when diagnosing whether improvements are broadly helpful rather than concentrated in a particular portion of the query set.

7.2 Choosing micro when overall volume matters

Micro averaging is appropriate when the application cares about total retrieval quality across all user interactions collectively—especially when queries differ greatly in the amount of relevant information available or when downstream consumers process pooled events (e.g., aggregating clicks or judged items). It emphasizes the question “Out of all retrieved items, how often are they relevant?”

In many evaluation reports, it is informative to present both macro-style and micro-style variants when feasible. Doing so clarifies whether a system’s apparent advantage is due to improvements on a specific subset of queries or due to consistent gains across the whole set. When space is limited, the choice should be justified by the intended interpretation: per-query performance versus pooled event relevance.

7.4 Dataset diagnostics to guide the choice

Before selecting an averaging rule, analysts can inspect:

  • the distribution of the number of relevant items per query,
  • the presence and frequency of queries with empty relevance sets,
  • the variance of per-query metric values for candidate systems,
  • whether some query categories dominate counts.

If the dataset shows strong heterogeneity in relevance support, macro and micro can yield meaningfully different rankings, and reporting both can prevent misinterpretation.

8 Implementation considerations

8.1 Computing per-query statistics efficiently

A practical approach is to compute all per-query components needed for the chosen metric (e.g., TP/FP/FN at cutoff \(k\), or AP components) in a loop over queries, then aggregate. For efficiency, implementations often vectorize relevance checks over retrieved lists, cache ideal DCG values for NDCG when relevance sets are fixed, and avoid recomputing denominators repeatedly.

8.2 Aggregation pitfalls (weighting bugs, normalization errors)

Common sources of error include:

  • mixing macro and micro logic (e.g., averaging per-query numerators/denominators rather than pooling counts),
  • using incorrect denominators for cutoff-based recall,
  • off-by-one mistakes in rank computations for MRR and AP,
  • normalizing after pooling with the wrong scale.

Careful unit tests with small synthetic relevance patterns help validate that the aggregation matches the intended mathematical definition.

8.3 Dealing with empty or undefined metrics

8.3.1 Skipping undefined queries vs assigning defaults

When a query’s metric is undefined (for example, recall when the query has zero relevant items), implementations must follow an explicit policy. A common convention is to skip undefined queries for the affected metric and renormalize the average over the remaining defined queries; another is to assign a default value based on a defined convention. Either way, the policy affects macro results because each query is a unit of influence. Micro results may be less impacted if pooled denominators remain nonzero, but undefined components must still be handled consistently to avoid silent errors.

8.4 Reproducibility and reporting details

Reproducible evaluation requires reporting:

  • the aggregation type (macro or micro),
  • cutoff values for precision/recall metrics,
  • how empty or undefined cases were treated,
  • the exact formula used for derived metrics like F1.

These details allow others to replicate results and compare systems fairly.

9 Worked examples

9.1 Small toy example illustrating macro vs micro

Consider two queries evaluated at the same cutoff \(k\). Suppose precision@\(k\) values are:

  • Query A: \(P@k=0.0\) (no retrieved items are relevant)
  • Query B: \(P@k=1.0\) (all retrieved items are relevant)

Macro precision is the average of per-query precisions: \[ P@k_{\text{macro}} = (0.0 + 1.0)/2 = 0.5. \] Micro precision depends on pooled counts. If both queries retrieve \(k\) items, micro precision also becomes 0.5. Divergence arises when the effective counts differ (for instance, if relevance judgments or evaluated support differ across queries, or if the evaluation procedure varies what is counted).

9.2 Effect of different relevant set sizes

Now consider two queries with different numbers of relevant documents. Suppose a system retrieves some relevant items from both queries, but misses many relevant items in the large-support query. Macro recall averages the per-query recall rates, so missing a large number of items on one query can be partially “compressed” into a single low recall value. Micro recall, however, accumulates false negatives across queries, so missing many relevant items in the large-support query strongly lowers the pooled recall. This illustrates why micro scores often reflect performance on the queries that contain most of the relevant evidence.

9.3 Comparing systems under each averaging scheme

Imagine System 1 improves many events on large-support queries but leaves several small-support queries nearly unchanged. System 2 gives a moderate gain across all queries, raising per-query metric values more uniformly. Under macro aggregation, System 2 may score better due to consistent per-query improvements. Under micro aggregation, System 1 may win because its gains contribute more pooled true positives and fewer pooled errors. The comparison thus depends on whether the evaluation target is “typical query quality” (macro) or “overall event relevance” (micro).