1 Introduction to F1 score

1.1 What the F1 score measures

The F1 score is a single-number metric for assessing the quality of a classification model, particularly for tasks where the positive class is rare or where errors have asymmetric consequences. It reflects how well a system identifies positive instances while also capturing most of the true positives.

1.2 When F1 score is used

F1 score is widely used in machine learning and information retrieval when the evaluation goal centers on the balance between catching relevant items and avoiding incorrect “positive” predictions. It is common in text classification, document retrieval, spam detection, and other settings where false positives and false negatives both matter.

1.3 Relationship to precision and recall

The F1 score aggregates precision and recall into one value. Precision emphasizes the reliability of predicted positives, while recall emphasizes coverage of actual positives. The F1 score is designed to reward models that perform well on both aspects rather than excelling at only one.

2 Mathematical definition

2.1 Precision and recall formulas

For binary classification, define:

  • Precision = TP / (TP + FP)
  • Recall = TP / (TP + FN)

Here, TP denotes true positives, FP false positives, and FN false negatives.

2.2 Harmonic mean interpretation

The F1 score is defined as the harmonic mean of precision and recall:

  • F1 = 2 · (Precision · Recall) / (Precision + Recall)

Using the harmonic mean makes the metric sensitive to low values: if either precision or recall is small, the F1 score drops accordingly.

2.3 General Fβ score and the special case β = 1

More generally, the Fβ score weights recall and precision differently:

  • Fβ = (1 + β²) · (Precision · Recall) / (β² · Precision + Recall)

When β = 1, recall and precision receive equal weight, yielding the F1 score.

2.4 Confusion matrix notation for F1

In confusion-matrix terms, the F1 score for the positive class depends only on TP, FP, and FN. True negatives do not directly affect the computation, which is a key reason F1 can remain informative even when negative classes dominate.

3 Computing F1 score in practice

3.1 Binary classification workflow

A typical workflow is:

1 Introduction to F1 score

2 Mathematical definition

3 Computing F1 score in practice

4 Interpreting results

The choice of decision rule (often a threshold on a probability score) can strongly influence the resulting value.

3.2 Handling edge cases (zero precision or zero recall)

If a model predicts no positives, then TP + FP = 0 and precision is undefined in the literal formula. Similarly, if there are no actual positives, TP + FN = 0 and recall is undefined. Many software libraries define precision or recall as 0 in such cases to ensure the metric is computable, though exact conventions can vary. This affects F1 directly because it depends on both quantities.

3.3 Example calculations

Suppose a model yields TP = 40, FP = 10, FN = 20:

  • Precision = 40 / (40 + 10) = 0.80
  • Recall = 40 / (40 + 20) = 0.67
  • F1 = 2 · (0.80 · 0.67) / (0.80 + 0.67) ≈ 0.73

This illustrates that even with strong precision, moderate recall limits the combined score.

3.4 Macro, micro, and weighted averaging

For multiclass classification, F1 can be extended using averaging schemes:

  • Macro F1: compute F1 for each class and take the unweighted mean.
  • Micro F1: pool TP, FP, and FN across classes before computing precision/recall.
  • Weighted F1: average per-class F1 using class frequencies as weights.

Macro averaging treats every class as equally important, micro averaging emphasizes overall performance, and weighted averaging lies between these extremes.

4 Interpreting results

4.1 How to read an F1 value

An F1 score near 1 indicates that predicted positives align closely with true positives and that most actual positives are recovered. Values near 0 suggest the model either misses many true positives, predicts too many false positives, or both.

4.2 Comparing models with F1

When models are evaluated on the same dataset with the same positive-class definition and decision threshold, comparing F1 scores provides a practical way to judge which model balances precision and recall more effectively. However, comparisons are less reliable when preprocessing differs or when the evaluation protocol changes.

4.3 Threshold effects on F1

If the classifier outputs probabilities, F1 depends on the threshold used to determine predicted labels. Raising the threshold often increases precision while decreasing recall, and lowering it does the opposite. Because the harmonic mean penalizes imbalance, the optimal threshold for F1 typically is not the default 0.5 and may need tuning.

4.4 Trade-offs between precision and recall

F1 can be viewed as a compromise: a model with higher precision but lower recall may still lose to one with more balanced performance. Conversely, improving the weaker of precision or recall can yield a larger increase in F1 than improving only the already-strong component.

5.1 F1 vs accuracy

Accuracy measures the proportion of all correct predictions, including true negatives. In highly imbalanced problems, a model can achieve high accuracy by mostly predicting the majority class, even if it performs poorly on the positive class. F1 avoids that particular failure mode because it focuses on TP, FP, and FN for the positive class.

5.2 F1 vs ROC-AUC and PR-AUC

ROC-AUC evaluates ranking quality across thresholds using true positive rate versus false positive rate. PR-AUC uses precision versus recall, typically offering more insight in tasks with class imbalance. F1 provides a single threshold-dependent summary, while ROC-AUC and PR-AUC reflect performance across many potential thresholds.

5.3 Precision–recall curves and choosing thresholds

Precision–recall curves visualize how precision changes as recall increases when the decision threshold varies. Selecting a threshold can be guided by where the curve best supports the desired balance, often related to maximizing F1 or meeting a target recall or precision constraint.

5.4 Use with multilabel classification (overview)

In multilabel settings, each instance can belong to multiple classes simultaneously. F1 is computed per label and then aggregated using macro, micro, or weighted averaging. Aggregation choices determine whether rare labels receive equal treatment (macro) or whether frequent labels dominate (micro), affecting interpretation.

6 Implementation and tooling

6.1 Common API usage in ML libraries

Most mainstream ML libraries provide built-in functions to compute F1 score. Typical inputs include:

  • true labels
  • predicted labels or predicted probabilities plus a threshold
  • averaging mode (binary, micro, macro, weighted)
  • handling of undefined metrics when denominators are zero

These tools aim to follow standard metric definitions, but the handling of edge cases may follow library-specific defaults.

6.2 Cross-validation and aggregation

In experiments with cross-validation, F1 can be reported per fold and then averaged. Aggregating across folds helps estimate how stable the metric is under different training/test splits. Reporting the distribution (e.g., mean and standard deviation) can provide more information than a single averaged number.

6.3 Reporting conventions in experiments

Clear reporting usually includes:

  • dataset name and split protocol
  • whether F1 is binary or multiclass/multilabel
  • averaging method (macro, micro, weighted)
  • the decision threshold used (if applicable)
  • whether results are from cross-validation or a fixed test set

Without these details, F1 values may not be comparable across studies.

6.4 Reproducibility considerations

Reproducibility depends on consistent preprocessing, deterministic data splits (where feasible), and consistent label definitions. For thresholding, it is also important to document how thresholds are selected—fixed ahead of time or tuned on a validation set—to avoid optimistic evaluation.

7 Limitations and pitfalls

7.1 Class imbalance behavior

Although F1 is often preferred over accuracy for imbalanced problems, it can still be influenced by imbalance through how averaging is done. Weighted and micro averages can be dominated by majority classes, while macro averages may be unstable for classes with few positive examples.

7.2 When a single score can mislead

A single F1 value compresses precision and recall into one number. Two models can share the same F1 score while having very different error profiles, such as one with higher precision but lower recall and another with the reverse. Examining both precision and recall alongside F1 can clarify such cases.

7.3 Sensitivity to decision thresholds

Because F1 depends on predicted labels rather than raw scores, small threshold changes can move the metric appreciably. This sensitivity is especially noticeable when precision–recall curves have steep regions.

7.4 Comparing datasets or label definitions fairly

Comparing F1 across datasets requires matching positive-class definitions, labeling guidelines, and evaluation pipelines. If one dataset labels borderline cases differently, the apparent performance gap may reflect annotation differences rather than model quality.

8.1 Information retrieval terminology

In information retrieval, precision and recall correspond to how effectively a system retrieves relevant documents. Precision aligns with the trustworthiness of returned items, while recall aligns with completeness. The F1 score is then a convenience measure for the combined retrieval effectiveness.

8.2 Learning from contingency tables

F1 is derived from the entries of a contingency table (TP, FP, FN, and typically implicit TN). It is an example of how performance measures can be computed from summary counts rather than from continuous outputs.

8.3 Cost of errors and metric alignment

Different applications assign different costs to false positives and false negatives. While F1 treats precision and recall symmetrically (or via Fβ when asymmetric), real-world cost structures may not match this assumption. Aligning the metric with the underlying error costs—through threshold choice or alternative Fβ weighting—can improve evaluation usefulness.