1 Reliability diagram basics

1.1 Purpose in probabilistic prediction

A reliability diagram is used to evaluate whether a probabilistic model’s predicted confidence corresponds to the true likelihood of events. Rather than judging predictions only by ordering or accuracy, it examines the relationship between the model’s stated probabilities and the empirical outcomes observed in data.

1.2 Calibration vs. discrimination

Calibration describes how well predicted probabilities match observed frequencies. Discrimination describes how well a model separates cases where an event occurs from those where it does not, often measured by ranking-based criteria. A model may rank cases effectively (good discrimination) yet still output probabilities that are systematically too high or too low (poor calibration). Reliability diagrams focus on calibration.

1.3 Terminology and key concepts

1.3.1 Predicted probability

A predicted probability is the model’s confidence that a particular event will occur for a given input. In binary settings, this is commonly the estimated probability that the positive class is the outcome.

1.3.2 Observed event rate

The observed event rate for a group is the fraction of samples in that group in which the event actually occurs. This empirical frequency provides the ground truth reference needed to assess calibration.

1.3.3 Probability binning

Probability binning partitions predictions into intervals (bins) over the probability range. Each bin aggregates samples whose predicted probabilities fall within a specified range, enabling the diagram to compare average prediction strength with realized frequency.

2 Construction of a reliability diagram

2.1 Data preparation

2.1.1 Defining events and outcomes

To build a reliability diagram, the analyst must specify the event of interest (e.g., whether a binary label indicates occurrence). For each sample, the dataset provides the model’s predicted probability and the corresponding true outcome label.

2.1.2 Handling missing predictions

Reliability diagrams typically require a predicted probability for each evaluated sample. If predictions are missing, samples can be removed or imputed, depending on the workflow. Removing missing predictions changes the evaluation set, while imputation requires additional assumptions and can distort calibration assessment.

2.2 Binning strategies

2.2.1 Fixed-width bins

Fixed-width binning divides the probability interval \([0,1]\) into equal-sized ranges (for example, 10 bins of width 0.1). This approach is simple and interpretable, but some bins may receive few samples, especially when the model outputs probabilities concentrated in limited regions.

2.2.2 Quantile-based bins

Quantile-based binning chooses bin boundaries so each bin contains approximately the same number of samples. This improves stability of empirical frequency estimates in sparsely populated regions, although it makes the meaning of each bin depend on the distribution of predictions in the evaluation set.

2.3 Computing bin statistics

2.3.1 Bin mean predicted probability

For each bin, the diagram typically uses the average predicted probability of samples in that bin as the x-value (or sometimes as a marker location). Using the mean aligns the displayed calibration point with the center-of-mass of predictions in that group.

2.3.2 Bin empirical frequency

The y-value is computed as the observed event rate within the bin: the number of positive outcomes divided by the total number of samples in that bin. This provides the empirical probability corresponding to the predictions gathered there.

2.3.3 Sample sizes and weighting

Bin sizes affect the reliability of the empirical frequency. Many implementations track counts to enable weighting or to display uncertainty. When summarizing calibration quantitatively, bins may be weighted by their sample counts to reflect statistical confidence.

2.4 Plot elements

2.4.1 Calibration curve (empirical vs. predicted)

The calibration curve connects, for each bin, the bin mean predicted probability (x-axis) with the bin empirical frequency (y-axis). Deviations from a 1:1 relationship reveal miscalibration.

2.4.2 Reference line for perfect calibration

A reference line indicates perfect calibration, typically the diagonal where predicted probability equals observed event rate. If the curve lies on this line across bins, the model’s probabilities match observed frequencies.

2.4.3 Error bars and uncertainty displays

Because each bin’s observed frequency is estimated from finite data, analysts may include uncertainty intervals (often derived from binomial approximations or resampling). These intervals help distinguish real calibration issues from sampling noise.

3 Interpreting reliability diagrams

3.1 Perfect calibration characteristics

Perfect calibration appears as the calibration curve coinciding with the reference diagonal. This means that across the probability spectrum, the model’s stated confidence agrees with the empirical event frequencies.

3.2 Underconfidence patterns

Underconfidence occurs when predicted probabilities are too low relative to observed frequencies. On the diagram, the curve tends to lie below the reference line in the relevant probability region, indicating events occur more often than the model expects.

3.3 Overconfidence patterns

Overconfidence occurs when predicted probabilities exceed observed frequencies. Visually, the calibration curve rises above the reference line, suggesting that the model is more certain than outcomes warrant.

3.4 Systematic bias and nonlinearity

If the curve deviates from the diagonal in a consistent, non-uniform way, it indicates systematic mismatch beyond simple linear scaling. For example, a curve that is S-shaped can suggest that probabilities are compressed in the mid-range or exaggerated at the extremes, reflecting nonlinear calibration error.

3.5 Sparse bins and visual artifacts

When bins contain few samples, empirical frequencies fluctuate more, which can create misleading wiggles in the curve. Uncertainty bars or strategies such as quantile binning and smoothing help mitigate these artifacts.

4.1 Expected Calibration Error (ECE)

Expected Calibration Error summarizes calibration deviation across bins, typically by averaging the absolute (or squared) difference between predicted probabilities and empirical frequencies. ECE depends on binning choices and is therefore most meaningful when accompanied by details of the evaluation setup.

4.2 Maximum Calibration Error (MCE)

Maximum Calibration Error captures the worst-case discrepancy across bins. Where ECE provides an overall sense of misalignment, MCE highlights whether any probability region exhibits severe calibration failure.

4.3 Brier score decomposition

The Brier score measures mean squared error between predicted probabilities and outcomes. It can be decomposed into calibration-related components and refinement-like components, linking the reliability diagram’s visual notion of calibration to a single numerical scoring rule.

4.4 Calibration curves vs. threshold-based metrics

Reliability diagrams differ from threshold-based metrics such as accuracy or F1 score, which depend on a chosen decision threshold and convert probabilities into hard predictions. Calibration quality concerns the probability values themselves, whereas threshold metrics reflect classification performance under a specific cutoff.

4.5 Reliability diagram variants for multiclass settings

In multiclass problems, reliability evaluation extends beyond a single probability. Common approaches compare the predicted confidence of the most likely class against empirical correctness frequency, using bins over confidence scores, or evaluate per-class reliability in one-versus-rest fashion. Diagram variants adapt the binning and aggregation to match the multiclass structure.

5 Extensions and best practices

5.1 Bin count selection and trade-offs

Choosing the number of bins affects resolution and stability. Too few bins can hide structured miscalibration, while too many bins can introduce noise due to small sample counts. A practical compromise aims to balance granularity with sufficient data per bin.

5.2 Smoothing and calibration curve stability

Smoothing can reduce fluctuations from limited data by replacing raw empirical points with a smoother estimate. While this can improve readability, it introduces design choices that may obscure abrupt calibration issues. Analysts often pair smoothing with uncertainty quantification.

5.3 Confidence intervals via resampling

Confidence intervals can be estimated through resampling methods such as bootstrapping. This approach accounts for sampling variability in both predicted probabilities and observed labels, yielding uncertainty bounds that can be plotted alongside the calibration curve.

5.4 Evaluation on validation vs. test sets

Calibration should be assessed on a dataset not used to fit the model parameters that produce the probabilities. When calibration methods are tuned, a separate validation set is commonly used for calibration fitting, while a final test set provides an unbiased estimate of calibration performance.

6 Calibration improvement methods (overview)

6.1 Post-hoc calibration approaches

Post-hoc calibration methods adjust predicted probabilities without changing the underlying model architecture. They typically learn a mapping from raw scores to calibrated probabilities using labeled data, aiming to reduce systematic miscalibration while preserving ranking relationships as much as possible.

6.2 Temperature scaling concepts

Temperature scaling transforms logits (or equivalent scoring values) by dividing them by a learned temperature parameter. The effect is to reshape probability distributions, commonly improving calibration when the base model’s errors are primarily due to over/under dispersion of confidences.

6.3 Isotonic regression concepts

Isotonic regression fits a monotonic function that maps uncalibrated scores to calibrated probabilities. Because it imposes only monotonicity, it can capture complex calibration curves, though it may overfit when the calibration dataset is small.

6.4 Platt scaling concepts

Platt scaling fits a logistic function to map model scores to probabilities. It is often used for binary classification and can provide a flexible yet structured recalibration, balancing adaptability with regularization inherent in the logistic form.

6.5 When recalibration may fail

Recalibration may not fully correct probability misalignment when the model’s uncertainty is not representable by a monotonic transformation, when labels are noisy, or when evaluation data differ substantially from the calibration data. In such cases, the reliability diagram may continue to show structured deviations even after post-hoc adjustment.