1 Foundations of Scoring Rules
1.1 Forecasts, outcomes, and scoring functions
A proper scoring rule evaluates a probabilistic forecast by assigning a numerical score after the eventual outcome is revealed. The forecasting object is typically a probability distribution over a set of possible outcomes, while the scoring function maps both the forecast and the realized outcome to a real number. The key modeling choice is whether the forecast space consists of unconstrained distributions (for example, any vector of probabilities summing to one) or a restricted family (for instance, distributions representable by a given model class).
1.2 Expected score and incentive compatibility
Because the score depends on what happens later, the forecaster cares about the expected value of the score under their own belief about future outcomes. Let the forecaster’s true belief distribution be the one they would use if asked directly. A scoring rule is designed so that, when the forecaster computes expected score using that belief, the best action is to report the same distribution. In this way, the rule acts as a mechanism translating subjective beliefs into incentive-compatible reports.
1.3 Proper versus strictly proper scoring rules
A scoring rule is proper if truthful reporting maximizes expected score for every forecaster belief. It is strictly proper if truthful reporting is the unique maximizer, except possibly on sets where different reports lead to identical expected scores. Strictness matters in applications where multiple probability distributions could otherwise tie, potentially allowing noise or strategic behavior to affect reported beliefs.
2 Mathematical Definition
2.1 Formal setup with probability distributions
Let the outcome space be a measurable set \(\mathcal{Y}\). Let \(\mathcal{P}\) denote a set of probability distributions on \(\mathcal{Y}\) that represent admissible forecasts. A scoring rule is a function \(S:\mathcal{P}\times \mathcal{Y}\to \mathbb{R}\), where \(S(p,y)\) is the score assigned if the forecaster reports distribution \(p\) and the realized outcome is \(y\). Under a belief \(q\in\mathcal{P}\), the expected score is \[ \mathbb{E}_{Y\sim q}[S(p,Y)]. \]
2.2 The propriety condition (truthful reporting)
The scoring rule is proper if for all \(q\in\mathcal{P}\), \[ \mathbb{E}_{Y\sim q}[S(q,Y)] \ge \mathbb{E}_{Y\sim q}[S(p,Y)] \quad \text{for all } p\in\mathcal{P}. \] Intuitively, each belief distribution \(q\) makes its own report optimal against the forecaster’s own expectation. This condition is the formal embodiment of incentive compatibility.
2.3 Uniqueness and strictness (when truthful is uniquely optimal)
The scoring rule is strictly proper if the inequality above is strict whenever \(p\neq q\) (under conditions that prevent pathologies like equal expected scores for distinct reports). When strictness holds, any deviation from the true belief reduces expected score, turning the incentive scheme into a strong identification principle for the reported distribution.
3 Construction Principles
3.1 Convex analysis and representation theorems
Many scoring rules can be constructed via convex analysis. A common theme is that proper scoring rules correspond to subgradients of convex functions (or related objects) defined on the forecast space. This perspective yields systematic methods: rather than guessing a scoring formula, one can start with a convex function that encodes the geometry of distributions and then derive a rule whose expected score is maximized precisely by truthful reports.
3.2 Relation to Bregman divergences
Bregman divergences provide a canonical bridge between convex functions and scoring rules. Given a strictly convex differentiable function \(\Phi\), the Bregman divergence measures how much \(\Phi\) increases when moving from one point to another using linearization at the second point. Proper scoring rules can be written so that the forecaster’s expected score difference equals (up to sign and constants) a Bregman divergence between the reported distribution and the belief. Because divergences are nonnegative, the incentive property follows naturally.
3.3 Generating scoring rules from entropy functions
Entropy-like quantities often serve as generating functions. For example, the logarithmic score is connected to Shannon entropy, while other entropies correspond to alternative scoring formulas. In this framework, the scoring rule is derived from how an entropy functional changes under perturbations of the reported distribution. Different entropies yield different notions of accuracy, sensitivity, and calibration behavior.
3.4 Discrete versus continuous outcome spaces
Construction methods must be adapted to whether \(\mathcal{Y}\) is finite, countable, or continuous. In discrete settings, distributions are probability vectors and scoring rules are often expressed componentwise. In continuous settings, densities may be used, but care is required: scoring rules based on densities can be sensitive to the choice of reference measure, and numerical evaluation often involves approximations. Properness can still be established, though the technical details depend on measurability and integrability conditions.
4 Canonical Examples
4.1 Logarithmic score
For a discrete outcome space with classes \(1,\dots,K\), the logarithmic score assigns \[ S(p,i)=\log p_i, \] where \(p_i\) is the probability reported for outcome \(i\). The expected logarithmic score is maximized when the reported probabilities match the belief distribution. This rule heavily penalizes assigning low probability to events that later occur, making it responsive to sharp forecasts.
4.2 Brier (quadratic) score
With the same discrete setup, the Brier score is \[ S(p,i)= -\sum_{k=1}^K (p_k-\mathbf{1}\{k=i\})^2, \] or equivalently defined up to an additive and multiplicative constant. It is based on squared error between the forecast probabilities and the one-hot encoding of the realized outcome. Unlike the logarithmic score, it penalizes overconfidence in a smoother, more uniformly distributed way.
4.3 Spherical score and geometric interpretations
The spherical score has a form related to cosine similarity between forecast probabilities and one-hot outcomes in an appropriate geometry. One common version for discrete outcomes is \[
| S(p,i)=\frac{p_i}{\|p\|}, |
|---|
\]
| where \(\|p\|\) is an \(\ell_2\)-norm. The normalization yields a geometric interpretation in terms of angles, and the corresponding properness arises from underlying convexity properties in transformed coordinates. |
|---|
4.4 Rank-based and transformation-based variants
Beyond straightforward functional forms, scoring rules can be built by transforming probability forecasts. Rank-based variants may depend on ordering information (or scores derived from it), while transformation-based variants apply a mapping to the forecast space before using a canonical proper scoring rule. These approaches can improve robustness or align scoring with application-specific priorities, though they require verification that properness is preserved under the chosen transformation.
5 Properties and Performance
5.1 Calibration and probabilistic accuracy
Proper scoring rules are closely connected to calibration: when forecasters are evaluated repeatedly, a scoring rule that incentivizes truthful reporting tends to produce forecasts whose empirical frequencies match predicted probabilities. Calibration, however, is not identical to having maximal score at every instance; it reflects long-run agreement between stated probabilities and observed outcomes.
5.2 Robustness to misspecification
In practice, models used to generate forecasts may be misspecified. Some scoring rules are more forgiving to model error than others because of how they weight discrepancies. Logarithmic scores typically amplify deviations where the model assigns near-zero probability to outcomes that occur. Quadratic-type rules often respond less violently, which can be advantageous when probabilities are only approximately correct.
5.3 Handling rare events and tail behavior
Rare outcomes reveal important differences between scoring rules. Logarithmic scoring can assign very large negative penalties when a forecaster understates the chance of an event that later occurs. This makes it effective for emphasizing tail risk but also sensitive to outliers and finite-sample estimation error. Alternative proper rules may temper this effect, offering different trade-offs between tail responsiveness and numerical stability.
5.4 Sensitivity to overconfidence and underconfidence
Proper scoring rules typically penalize both overconfidence and underconfidence, but with differing curvature. Strictly convex constructions yield strong penalties for sharp errors in probability assignment. For example, squared-error-based rules tend to distribute penalty according to Euclidean distance from the realized indicator, while log-type rules correspond to information-theoretic mismatch and therefore can react more strongly when the reported distribution is too concentrated.
6 Generalizations and Variants
6.1 Multiclass probability forecasting
For multiclass problems, proper scoring rules operate on probability vectors over \(K\) classes. A key requirement is that the scoring rule be well-defined on the simplex (including boundary points with zero probabilities, depending on the chosen rule). Extensions often preserve strict properness on the interior of the simplex, while boundary behavior can influence practical implementation.
6.2 Multivariate and structured prediction settings
Structured outcomes—such as sequences, graphs, or sets—require forecasts that capture dependencies across components. Proper scoring rules can be defined over structured probability distributions, but computational considerations become central: evaluating expected scores may require summations over large state spaces or approximations such as sampling and variational inference.
6.3 Proper scoring rules for conditional forecasts
Conditional forecasting concerns predictions given observed features \(x\). A conditional scoring rule evaluates a forecast \(p(\cdot\mid x)\) after observing the outcome \(y\) associated with that context. Properness can be extended so that, for each context, reporting the true conditional distribution maximizes conditional expected score. This yields objectives used in probabilistic learning with covariates.
6.4 Local versus non-local scoring rules
Some scoring rules are local, meaning the score for outcome \(y\) depends only on the forecast’s probability at \(y\) (and not on the entire distribution). Others are non-local, depending on multiple components jointly. Local rules can be simpler and computationally efficient, while non-local rules may capture interactions between probabilities across outcomes and can lead to different calibration dynamics.
7 Scoring Rules in Decision Theory and Statistics
7.1 Connection to Bayes risk and optimal decisions
In decision theory, an agent chooses actions to maximize expected utility or minimize expected loss. Proper scoring rules connect to Bayes risk by treating probabilistic forecasts as strategies and the scoring function as a loss (or negative utility). Under appropriate assumptions, the expected score under belief maximization aligns with optimal decision-making, providing a principled way to select scoring functions that reflect the agent’s informational needs.
7.2 Elicitation of beliefs from experts
Proper scoring rules can be used as mechanisms to elicit probabilistic beliefs from human experts or systems. By rewarding forecasts according to realized outcomes, the evaluator encourages participants to report their genuine beliefs rather than convenient values. In practical elicitation, one often needs to specify what constitutes an admissible forecast, how ties or boundary events are handled, and how to mitigate incentives for strategic manipulation.
7.3 Estimation objectives induced by scoring rules
When a probabilistic model is trained by minimizing expected negative scores, the scoring rule induces an estimation objective. For example, maximizing expected logarithmic score corresponds to likelihood-based training under common modeling assumptions. Quadratic scores correspond to mean squared error-like objectives in probability space. This viewpoint unifies probabilistic forecasting and machine learning under a common principle: different scoring rules correspond to different statistical targets and learning dynamics.
8 Practical Considerations
8.1 Numerical stability and implementation details
Implementation requires attention to floating-point issues. Logarithmic scores can suffer from \(\log 0\) when a model assigns zero probability; numerical stabilization typically involves smoothing or bounding probabilities away from zero. Quadratic scores are usually more stable but still require careful handling when probability vectors are large-dimensional or when normalization is approximate.
8.2 Choice of scoring rule for different applications
Selecting a scoring rule depends on what errors are most costly. If accurate representation of event likelihoods is crucial, log-type scores may be preferred due to their strong information sensitivity. If a smoother penalty is desired to reduce the influence of extreme miscalibration, quadratic scores or other bounded proper rules may be more appropriate. The choice also affects interpretability: scores can be presented as losses, per-outcome penalties, or aggregated metrics.
8.3 Handling constraints (e.g., probability simplex issues)
Forecasting models may output values that do not lie exactly on the simplex due to approximation error, model structure, or optimization. Practical pipelines often apply normalization (e.g., softmax transforms) or project onto the feasible set. Any projection step must be consistent with how the scoring rule interprets forecasts; otherwise, the theoretical guarantees of properness can be weakened in practice.
8.4 Common pitfalls and interpretation of scores
Scores are not directly comparable across different scoring rules without calibration of scale and convention (since many scores are defined up to affine transformations that preserve properness). Users may also misinterpret average scores when the underlying distribution is drifting across time or contexts. Another pitfall is evaluating a model without accounting for censoring, missing data, or changes in the outcome-generating process, which can distort the intended meaning of calibration and accuracy.
9 Theoretical Results
9.1 Characterizations and equivalence of classes
A major body of theory characterizes the set of all proper scoring rules under suitable assumptions. Many characterizations show that proper scoring rules correspond to convex functions and that two scoring rules may be equivalent in the sense that they differ only by transformations that preserve the optimizer of expected score. These results clarify which aspects of a scoring rule affect incentives and which merely shift or rescale reported values.
9.2 Impossibility or limitations in certain settings
Proper scoring rules can fail to exist in full generality when forecast spaces are too restricted or when regularity conditions (such as differentiability or measurability) break down. Even when proper scoring rules exist, strict properness may only hold on subsets of the forecast space, such as the interior of the simplex. Additionally, in continuous settings, improper use of densities relative to reference measures can lead to scoring that is not genuinely proper.
9.3 Relationships to other divergence measures
Because expected score differences can be expressed via divergences, scoring rules relate naturally to information measures. Depending on the generating function, divergences may align with Kullback–Leibler-type quantities, squared distances in appropriate coordinates, or more general f-divergences and Bregman-type divergences. These relationships help translate theoretical properties—such as nonnegativity and equality conditions—into incentive and calibration conclusions.
10 Applications in Modern Systems
10.1 Machine learning for probabilistic outputs
Many machine learning systems output probabilities rather than point predictions, including classifiers with softmax layers and regression models producing predictive distributions. Training with losses derived from proper scoring rules yields probabilistic calibration objectives and often improves the quality of uncertainty estimates. The scoring rule choice influences gradient behavior, robustness, and the model’s tendency to produce concentrated versus diffuse predictions.
10.2 Weather, risk, and forecasting dashboards
Operational forecasting products often display predicted probabilities for events such as precipitation thresholds or safety-related alerts. Proper scoring rules provide a principled way to evaluate these forecasts over time, enabling performance comparisons across models and methods. Dashboard metrics based on proper scores can support iterative model improvement and accountability in forecasting workflows.
10.3 Online learning and adaptive prediction
In online settings, forecasters update predictions as data arrive. Proper scoring rules supply feedback signals that are consistent with truthful probabilistic reporting, even when models adapt. This helps formalize regret bounds and convergence behavior under different learning algorithms, provided the scoring rule’s properties match the online learning framework.
10.4 A/B testing with probabilistic metrics
Experimentation platforms sometimes need metrics beyond click-through rates, such as the quality of probabilistic rankings or uncertainty-aware predictions. Proper scoring rules allow A/B comparisons using probabilistic outputs by converting realized outcomes into score-based measures. Because many scoring rules are sensitive to calibration and tail events, they can distinguish systems that achieve similar point-level accuracy but produce different uncertainty estimates.