1 Definition and Intuition

Macro-F1 (macro-averaged F1-score) is a classification evaluation metric that computes an F1-score for each class separately and then averages those per-class scores. The key idea is that every class contributes equally to the final result, regardless of how many instances of each class appear in the dataset.

1.1 Precision, Recall, and F1 for a Single Class

For a particular class, F1 combines two complementary quantities:

  • Precision measures how many predicted instances of the class are actually correct.
  • Recall measures how many true instances of the class are successfully retrieved by the model.
  • F1-score is the harmonic mean of precision and recall, providing a single value that balances both aspects.

When viewed per class, F1 captures the trade-off between “over-predicting” a class and “missing” it.

1.2 Averaging Strategies: Macro vs Micro

Averaging determines how per-class performance is summarized:

  • Macro-F1 averages per-class F1-scores with equal weight.
  • Micro-F1 aggregates counts (e.g., true positives, false positives, false negatives) across classes before computing the final score.

Micro-F1 tends to reflect performance on frequent classes more strongly, while macro-F1 maintains parity among classes.

1.3 When Macro-F1 Is Preferable

Macro-F1 is often preferred when:

  • The dataset is class-imbalanced, and a small minority class should not be ignored.
  • The evaluation goal is to ensure reasonable behavior across categories, not only on the most common one.
  • Stakeholders want an overall score that treats each label as equally important.

In such settings, improvements on minority classes can materially raise the macro-F1 score, even if majority-class performance changes little.

1.4 Common Pitfalls and Misinterpretations

Several misunderstandings occur in practice:

  • High macro-F1 does not guarantee uniform quality: a model can do well overall while still failing on specific classes with low support if those classes have different error profiles.
  • Class importance may be misrepresented: equal weighting is a design choice; if some classes are truly more critical, unweighted averaging may not match the real objective.
  • Ignoring label prevalence: macro-F1 can appear modest even when the model is accurate for most examples, because errors on rare labels carry full weight.

2 Mathematical Formulation

Macro-F1 is grounded in the computation of precision and recall per class, which in turn depend on class-specific confusion matrix components.

2.1 Per-Class Confusion Components (TP, FP, FN)

For a given class \(k\), define:

  • True Positives (TP\(_k\)): instances of class \(k\) predicted as \(k\)
  • False Positives (FP\(_k\)): instances not in class \(k\) predicted as \(k\)
  • False Negatives (FN\(_k\)): instances of class \(k\) predicted as not \(k\)

These values treat class \(k\) as the “positive” class while collapsing all others into the negative set.

2.2 Per-Class F1-Score Computation

Precision and recall for class \(k\) are:

\[ \text{Precision}_k = \frac{TP_k}{TP_k + FP_k}, \quad \text{Recall}_k = \frac{TP_k}{TP_k + FN_k} \]

Then the per-class F1-score is:

\[ \text{F1}_k = \frac{2 \cdot \text{Precision}_k \cdot \text{Recall}_k}{\text{Precision}_k + \text{Recall}_k} \]

Equivalently, using counts:

\[ \text{F1}_k = \frac{2TP_k}{2TP_k + FP_k + FN_k} \]

2.3 Macro Averaging Across Classes

If there are \(K\) classes, macro-F1 is computed as the unweighted mean of the per-class scores:

\[ \text{Macro-F1} = \frac{1}{K}\sum_{k=1}^{K} \text{F1}_k \]

This formulation makes explicit that each class contributes with the same coefficient \(1/K\).

2.4 Handling Undefined F1 Values (No Positives)

Per-class precision or recall can become undefined in edge cases, for example when:

  • \(TP_k + FP_k = 0\): the model never predicts class \(k\)
  • \(TP_k + FN_k = 0\): the dataset contains no true instances of class \(k\)

Different libraries adopt different conventions, such as treating the corresponding F1 as 0, skipping that class, or raising a warning. For reproducible evaluation, it is important to align with the implementation’s policy for “undefined” cases.

3 Interpretation in Practice

Interpreting macro-F1 requires understanding what the metric emphasizes and what it may obscure.

3.1 What a Macro-F1 Score Communicates

A macro-F1 value summarizes average per-class balance between precision and recall. Because the metric is computed separately for each label, it reflects how well the model recognizes each category while managing false alarms for that category.

In effect, it measures whether the classifier is consistently competent across labels rather than only on the most prevalent ones.

3.2 Sensitivity to Class Imbalance

Unlike micro-averaging, macro-F1 does not let frequent classes dominate the final number. Consequently:

  • Errors on minority classes can significantly lower macro-F1.
  • Gains on rare labels can raise macro-F1 even if overall accuracy stays similar.

This sensitivity makes macro-F1 useful for diagnosing whether a model is neglecting hard-to-learn categories.

3.3 Comparing Models Using Macro-F1

When comparing multiple models, macro-F1 supports fairer comparison under label imbalance, but only when the label set and evaluation split are the same. A higher macro-F1 suggests an overall improvement in per-class precision-recall trade-offs.

However, two models can share a similar macro-F1 while having different strengths across classes, so pairing macro-F1 with per-class reporting is typically informative.

3.4 Relationship to Error Patterns

Macro-F1 can be influenced by both types of error:

  • False positives for a class reduce precision and thus can lower that class’s F1.
  • False negatives for a class reduce recall and similarly depress the F1 for that label.

Therefore, changing the decision threshold, training objective, or calibration can alter macro-F1 by shifting which mistakes occur for each class.

4 Computation and Implementation

Accurate computation depends on label representation, library conventions, and the model’s output format.

4.1 Data Labeling Requirements (Single-label vs Multi-label)

Macro-F1 can apply in different problem types:

  • Single-label (multi-class): each instance belongs to exactly one class. Per-class TP/FP/FN are computed against the “one-vs-rest” view of each label.
  • Multi-label: each instance can belong to multiple classes simultaneously. Per-class TP/FP/FN are computed per label based on whether that label is present, still treating each label as its own binary classification problem.

In multi-label settings, macro-F1 averages across labels, often reflecting performance across all possible tags.

4.2 Computing Macro-F1 in Common Libraries

Most machine learning toolkits provide a macro-F1 option that:

  1. Computes per-class confusion statistics.
  2. Derives per-class F1.
  3. Averages across classes (unweighted).

Users should verify parameters related to averaging mode (macro vs micro), handling of absent classes, and whether labels are treated as multi-class or multi-label.

4.3 Thresholding and Decision Rules (for Probabilistic Outputs)

For models that output probabilities or scores, predicted labels depend on a decision rule. In multi-class classification, the class with the maximum score is chosen by default; in multi-label classification, one often applies per-label thresholds.

Macro-F1 can change substantially when thresholds are adjusted because precision and recall trade off differently for each class.

4.4 Efficiency Considerations for Large Class Sets

When the number of classes is large, computation and bookkeeping become more expensive:

  • Computing per-class TP/FP/FN may require maintaining large confusion structures.
  • Macro-F1 may need careful implementation to avoid unnecessary memory overhead.

Practical solutions include using sparse representations for labels when many classes are rarely present and leveraging optimized library implementations.

5 Evaluation Protocols

How macro-F1 is reported depends on the experimental design and the granularity of reporting.

5.1 Train/Validation/Test Splits and Macro-F1

Macro-F1 should be computed on a held-out set (validation for tuning, test for final reporting) to reduce the risk of overfitting to the evaluation metric. When class imbalance exists, keeping class proportions stable across splits is helpful, though not always possible.

5.2 Cross-Validation with Macro-F1

Cross-validation provides a more robust estimate by evaluating the model on multiple folds. With macro-F1:

  • One can compute macro-F1 per fold and then average across folds.
  • Alternatively, some workflows aggregate predictions across folds before computing a single macro-F1, though this can differ slightly depending on library and implementation choices.

5.3 Reporting Per-Class and Aggregate Scores

Because macro-F1 is an average, it is useful to report:

  • The overall macro-F1 value
  • The per-class F1-score vector
  • Optionally class-wise precision and recall

This helps distinguish whether an improvement is broad-based or driven by a subset of labels.

5.4 Calibration vs Ranking Metrics

Macro-F1 is primarily a classification accuracy-type metric that depends on the decision rule, not on probabilistic calibration directly. Metrics used for calibration (e.g., calibration curves) or ranking quality (e.g., certain AUC variants) capture different properties.

Nevertheless, better-calibrated models can enable thresholding strategies that improve macro-F1, particularly in multi-label scenarios.

6 Variants and Extensions

Extensions of macro-F1 adjust weighting, scope, or the averaging dimension.

6.1 Macro-F1 in Multi-Class Classification

In multi-class classification, macro-F1 treats each class as a separate one-vs-rest binary problem and averages the resulting F1-scores. This yields a single score reflecting label-wise balance across the entire category set.

6.2 Macro-F1 in Multi-Label Classification

In multi-label tasks, macro-F1 averages across labels rather than across exclusive classes. Each label’s TP/FP/FN are computed based on which instances include that label, and the harmonic mean balances precision and recall per label.

This can be more sensitive to threshold selection because each label may require different operating points.

6.3 Weighted-F1 vs Macro-F1

A common variant is weighted-F1, which averages per-class F1-scores using weights proportional to class frequency. This reduces the impact of rare classes compared with macro-F1, often aligning more closely with overall dataset composition.

Macro-F1 and weighted-F1 therefore represent two different notions of “overall performance,” one unweighted and one frequency-aware.

6.4 “Macro” over Subsets of Classes

Some evaluation designs compute macro-style averaging over a subset of labels, such as:

  • Only the labels present in a particular domain subset
  • Only the “critical” labels defined by product requirements
  • Labels above a minimum support threshold

This changes the interpretation: the resulting score measures balanced performance over the selected set rather than over all classes.

7 Example Workflows

Macro-F1 is commonly embedded into end-to-end model development pipelines.

7.1 Baseline Classifier Evaluation

A typical starting point is training a baseline model and computing macro-F1 on a validation set. The per-class F1 scores identify which categories are most problematic, while the aggregate macro-F1 provides a single benchmark for later comparisons.

7.2 Hyperparameter Tuning Using Macro-F1

During tuning, a workflow can select hyperparameters that maximize validation macro-F1. This can be especially helpful when optimizing for equitable label performance under imbalance, such as improving minority-class recall without letting false positives explode for those classes.

7.3 Error Analysis Guided by Per-Class F1

After training, analysts can inspect classes with low F1 to guide further investigation. Low precision suggests frequent false alarms, while low recall indicates that true instances are being missed. This information can inform targeted actions such as rebalancing data, refining features, or adjusting thresholds.

7.4 Model Selection Criteria and Trade-offs

Choosing the “best” model based on macro-F1 may trade off against metrics that emphasize majority classes or overall accuracy. A model with slightly higher macro-F1 might have lower performance on a dominant class but much better balance elsewhere. Consequently, selection criteria should match the intended application objective, and macro-F1 should be considered alongside per-class results and other relevant metrics.