1 Definition and Intuition

1.1 What counts as a scoring rule

A proper scoring rule is a function that assigns a numerical score to a probabilistic forecast once the realized outcome is observed. Formally, the forecast is a probability distribution over possible outcomes, and the scoring rule maps that distribution and the realized outcome to a payoff (which can be interpreted as either a “score” to maximize or a “loss” to minimize, depending on sign conventions). The defining feature is not how large the scores are, but how the expected score behaves under different choices of forecast distributions.

In common usage, “better” forecasts receive higher expected scores when they correctly reflect the forecaster’s underlying beliefs about the outcome-generating process.

1.2 Expected score and incentives

Suppose an agent’s true belief about the outcome is a probability distribution. The score they receive depends on what distribution they announce. The expected score is the belief-weighted average of the realized-outcome scores. A scoring rule is “incentive-compatible” when announcing one’s own belief distribution maximizes this expected score. This creates a direct link between statistical accuracy and decision incentives: truthful reporting is optimal under the rule.

1.3 Propriety vs. strict propriety

Propriety means that truthful reporting is optimal in expectation, but not necessarily the unique optimizer. Strict propriety strengthens the requirement: the agent’s own belief distribution is the unique maximizer of expected score (typically up to equivalence classes in the presence of constraints). In practice, strictness affects how strongly a forecaster is pushed toward a single best report rather than a set of equally good reports.

1.4 Discrete vs. continuous outcome spaces

Outcome spaces can be discrete (finite or countable) or continuous. In discrete settings, a forecast is typically a probability mass function and the score evaluates directly at the realized category. In continuous settings, forecasts are probability densities or distributions over measurable sets; scoring rules must handle issues such as normalization, integrability, and how pointwise evaluation is defined. Some continuous-time or density-based scoring rules use concepts analogous to likelihood, while others score predictions via comparisons between distributions that avoid direct dependence on point values.

2 Formal Framework

2.1 Forecast distributions and outcome models

Let the outcome be an element of a space \(\mathcal{Y}\). A probabilistic forecast is a distribution \(P\) on \(\mathcal{Y}\). Often \(\mathcal{Y}\) is discrete with \(P(y)\) for each \(y\), or \(\mathcal{Y}\) is continuous with density \(p(y)\) relative to a reference measure. The realized outcome is a random variable \(Y\) distributed according to a “true” but possibly unknown distribution \(Q\). The scoring rule evaluates the pair \((P, y)\) where \(y\) is the realized outcome.

2.2 Scoring rule specification

2.2.1 Pointwise scores

A (proper) scoring rule can be represented by a function \(S(P, y)\) that returns a score when outcome \(y\) occurs under announced distribution \(P\). Larger values correspond to better forecasts when the rule is written in “score” form. In “loss” form, one often minimizes \(-S(P,y)\); most theoretical statements translate by a sign change.

For discrete outcome spaces, \(S(P,y)\) typically depends on the probability \(P(y)\) assigned to the outcome that actually occurred, possibly plus additional terms. For continuous spaces, many classical rules depend on how the forecast density assigns mass near the realized outcome, sometimes through log-density-like expressions.

2.2.2 Expected score functional

Given a belief distribution \(Q\) for the true data-generating process, the expected score when announcing \(P\) is \[ \mathbb{E}_{Y\sim Q}[S(P,Y)]. \] The propriety property is stated in terms of the maximization of this functional over the class of allowed forecasts. If \(P=Q\) maximizes the expected score, the rule is proper relative to the forecast class. Strict propriety requires that \(P=Q\) yields a strictly larger expected score than any other admissible distribution (with standard caveats about equivalence).

2.3 Conditions for propriety

Propriety can be analyzed through functional inequalities: for all true distributions \(Q\) in the relevant class, and all candidate forecasts \(P\) in that class, \[ \mathbb{E}_{Y\sim Q}[S(Q,Y)] \ge \mathbb{E}_{Y\sim Q}[S(P,Y)]. \] Proving such inequalities often uses convexity properties and representations via divergences (see later sections). When the forecast class is restricted (e.g., only certain parametric families), “relative propriety” may hold only within the restricted set.

2.4 Relationship to optimization and decision theory

From a decision-theoretic perspective, a scoring rule induces a Bayes risk: the expected loss minimized (or expected score maximized) given a belief distribution. Proper scoring rules are therefore a way to translate probabilistic beliefs into optimal decisions under uncertainty. They also connect to risk minimization: each rule provides a criterion that selects the forecast distribution best aligned with the agent’s beliefs in expectation.

In mechanism-design settings, the same incentive property is formalized as truthful reporting being a dominant or equilibrium strategy under appropriate assumptions.

3 Canonical Examples

3.1 Logarithmic (log) score

A common scoring rule in probabilistic forecasting is the logarithmic score: \[ S(P,y)=\log P(y) \] for discrete \(y\), or its density analogue in continuous spaces. When maximizing expected log score, an agent is rewarded in proportion to how likely the realized outcome was under their announced distribution. Under standard conditions, reporting the true belief distribution maximizes expected log score.

3.1.1 Interpretation via likelihood

The log score is closely related to likelihood: the term \(\log P(y)\) is the log-probability assigned to the realized observation. In statistical estimation and machine learning, maximizing sums of log scores over data points corresponds to maximizing likelihood under the forecast model. This gives an intuition for why log scoring encourages calibrated probability assignments: placing probability mass on realizations that actually occur yields higher expected reward.

3.2 Brier (quadratic) score

For a discrete outcome space with categories \(i\), one version of the Brier score is \[ S(P,y_i)= -\sum_j (P(j)-\mathbf{1}\{j=i\})^2 \] in score form (or equivalently a positive squared-error loss). It compares the announced probability vector to the one-hot encoding of the realized outcome.

3.2.1 Binary case

In the binary case with outcome \(Y\in\{0,1\}\), if a forecast assigns probability \(p\) to outcome \(1\), the squared error loss is \((p-Y)^2\). The expected loss under beliefs about \(Y\) is minimized by setting \(p\) equal to the belief probability. Thus the Brier score is proper, and often strictly proper when probabilities are identifiable.

3.2.2 Multiclass extension

For multiclass outcomes, the general squared-distance form penalizes deviation from the one-hot realized indicator. The multiclass Brier score retains properness: the expected score is maximized by announcing the belief distribution, because the squared distance is minimized in expectation by matching the belief probabilities.

Spherical-type scores arise from comparing normalized probability vectors using Euclidean-like geometry on probability simplices. One representative form is \[

S(P,y_i)= \frac{P(i)}{\|P\|},

\] or variants that differ by normalization constants. These rules are proper under appropriate definitions and parameterizations, and they often emphasize agreement between the direction of probability vectors rather than their absolute scale.

3.4 Continuous-measure examples

For continuous outcomes, straightforward analogues of discrete rules require care. One approach uses density-based scoring rules such as the log score with density values (relative to a reference measure). Another class uses scores derived from divergences between distributions, which compare the announced forecast distribution to the true distribution without requiring point evaluation to be well behaved. In continuous settings, properness depends on integrability and measurability conditions so that expected scores are finite and optimization is well defined.

4 Theory and Characterizations

4.1 Proper scoring rules and convexity

A central theoretical link is that many proper scoring rules correspond to convex functions on the space of distributions (or on related parameter spaces). The expected score often becomes a difference between a convex function evaluated at the announced distribution and a term involving its subgradient. Convexity supplies both existence of supporting hyperplanes and the inequality structure needed for propriety.

This convex-analytic viewpoint also clarifies why strict propriety is tied to differentiability or curvature conditions: when the associated function has enough curvature, the maximizer (or minimizer in loss form) becomes unique.

4.2 Subgradients and supporting hyperplanes

When the convex function is not differentiable, subgradients provide the relevant generalization of gradients. Expected score optimality can be expressed via inequalities that resemble “supporting hyperplane” conditions: \[ G(Q) \ge G(P) + \langle g_P, Q-P\rangle, \] with equality at \(Q=P\) under the right circumstances. Scoring rules can be constructed so that expected score differences correspond to the slack in these inequalities. This yields systematic ways to derive new proper rules from convex potentials.

4.3 Bayes risk and optimal forecasts

Under any proper scoring rule, the forecast that maximizes expected score with respect to a belief distribution \(Q\) is the Bayes-optimal forecast for that decision problem. Bayes risk (the optimized expected loss) therefore reflects the informativeness of the scoring rule: some rules produce Bayes risks with distinct sensitivity to different types of probability mismatch.

In decision theory terms, the agent’s belief is the “state of knowledge,” and the scoring rule functions as the payoff structure whose Bayes action recovers truthful beliefs.

4.4 Uniqueness up to transformations

A scoring rule may be altered by transformations that preserve incentives, such as adding terms depending only on the reported distribution in a way that does not change which forecast maximizes expected score. Characterizations typically describe equivalence classes of scoring rules that differ by such transformations. Consequently, two rules can be considered the “same” from the incentive perspective even if their numerical outputs differ by affine changes or normalization conventions.

5.1 Proper scoring rules and Bregman divergences

Many proper scoring rules admit representations in terms of divergences derived from convex functions. In particular, Bregman divergences measure the gap between a convex function and its linear approximation. Proper scoring rules can be constructed so that the expected score under the true distribution equals the negative of a Bregman divergence (up to constants and sign conventions). This provides a direct mathematical bridge: incentives favor forecasts that minimize divergence from the belief distribution.

5.2 Kullback–Leibler divergence connection

For the logarithmic score, the expected score difference is governed by the Kullback–Leibler divergence. In broad terms, maximizing expected log score is equivalent to minimizing the KL divergence between the announced distribution and the true one. This offers a clear interpretability: forecasts are rewarded in proportion to how efficiently they compress the outcomes according to the announced probabilities, relative to the true distribution.

5.3 Total variation and other divergence measures (high-level)

Not all proper scoring rules correspond to KL divergence. Some are linked to other measures of discrepancy, including divergences related to geometry of probability spaces. At a high level, different scoring rules emphasize different aspects of mismatch—such as how they treat tail probabilities or how they weigh errors across outcomes—leading to divergence measures with distinct sensitivity profiles.

5.4 Calibration implications via divergence minimization

Because proper scoring rules reward distributions that minimize divergence (in expectation), they often imply calibration-like behavior when used repeatedly or within model classes. If the forecaster can effectively adjust their announced probabilities, optimizing the expected scoring rule drives them toward forecasts that match the distribution generating the outcomes. This yields practical interpretability: improvements in score correspond to better probabilistic alignment, not merely better accuracy under a point prediction.

6 Multivariate and Structured Forecasting

6.1 Forecasts over multiple events

Many forecasting tasks involve multiple related outcomes. Examples include several binary events, categorical outcomes with dependencies, or complex systems with structured outcomes. A proper scoring rule can be extended to handle vector- or set-valued predictions by defining how the joint distribution over outcomes is scored once the full realization is observed.

6.2 Joint vs. marginal scoring

A key design choice is whether to score the joint distribution directly or to evaluate marginals separately. Joint scoring can capture dependence structure but may be more demanding to specify and compute. Marginal scoring evaluates each component distribution, often producing scores that reflect correct marginal probabilities but may not fully validate dependence.

Proper composite strategies can combine both approaches, but the choice affects what aspects of the probabilistic model are identified as necessary for maximizing expected score.

6.3 Conditional and sequential scoring

In sequential settings, forecasts are updated over time as new information arrives. Proper scoring rules can be used in a dynamic manner by scoring each forecast at the moment it is issued, then combining scores over time. With appropriate conditions, the resulting cumulative score encourages each conditional forecast to be truthful given the information available at that time.

6.4 Factorized scoring approaches

When the joint distribution factorizes according to a graphical model or assumption of conditional independence, scoring rules can be adapted to exploit this structure. Factorized scoring computes scores from components (such as conditional distributions) in a way that preserves properness for the factorized class. This can reduce computational burden while retaining incentive properties within the chosen model family.

7 Implementation Considerations

7.1 Handling zero probabilities and numerical stability

Some scoring rules, notably the log score, can produce \(-\infty\) when the announced distribution assigns zero probability to an event that occurs. Implementations therefore require smoothing, careful parameter constraints, or the use of bounded variants. Numerical stability also matters in continuous cases where densities can be extremely small; underflow or overflows can distort computed scores.

Practical implementations often operate with stabilized computations (e.g., log-sum-exp tricks) and enforce probability lower bounds consistent with the theoretical scoring rule being approximated.

7.2 Scoring with censored or partially observed data (overview)

Real datasets sometimes contain incomplete observations, such as censoring in survival analysis or missingness in covariates. Scoring in such contexts typically requires integrating the scoring function over unobserved outcomes conditional on available information. While the details depend on the data-generating model and the censoring mechanism, the general theme is that a proper rule can be adapted by averaging over the missing parts in a way consistent with conditional beliefs.

7.3 Choice of scoring rule and practical tradeoffs

Different scoring rules have different sensitivity characteristics. The log score strongly penalizes assigning low probability to events that happen, encouraging sharp distributions but risking instability with zeros. The quadratic/Brier family tends to be smoother and may be more robust to probability misspecification, yet it can be less punitive about extreme tail errors. Selection depends on the domain, expected calibration behavior, and the computational constraints of the forecasting system.

7.4 Robustness to model misspecification

Even when a scoring rule is proper relative to a full distribution class, real forecasting models may be restricted to parametric or structured families. In those cases, maximizing expected score yields the best-fitting member within the restricted class, not the true distribution. The degree to which the scoring rule identifies correct probabilities then depends on approximation capacity and identifiability. Robustness discussions also consider how the scoring rule responds when the true data distribution lies outside the model family.

8 Applications

8.1 Forecast evaluation in statistics and forecasting

Proper scoring rules are widely used to evaluate probabilistic forecasting models. Compared with accuracy of point predictions, they assess whether the forecasted uncertainty matches realized variability. In time series and general predictive modeling, score-based evaluation enables comparing competing models even when they differ in calibration and distributional assumptions.

8.2 Machine learning and probabilistic model assessment

In machine learning, scoring rules function as training objectives or evaluation metrics for probabilistic classifiers and density estimators. Log-score-like objectives are closely related to maximum likelihood training, while quadratic-type objectives correspond to regression-like error measures in probability space. Proper scoring provides theoretical justification that optimizing the objective encourages well-calibrated probability outputs under appropriate conditions.

8.3 Elicitation of expert beliefs (overview)

When experts are asked to provide probability forecasts, proper scoring rules can be used to convert subjective beliefs into incentive-compatible numerical reports. The expert’s expected reward is maximized by reporting their genuine belief distribution. This approach is particularly useful when direct verification of beliefs is impossible and only realized outcomes can be observed.

8.4 Ensemble learning and stacking (overview)

In ensemble and stacking frameworks, proper scoring rules can guide how to combine multiple probabilistic models. If the combination method is chosen to maximize expected proper score, it can yield improved calibration and predictive performance. Some stacking procedures can be interpreted as optimizing a proper score over mixture weights or model outputs, aligning ensemble training with probabilistic fidelity.

9 Strict Propriety and Identifiability

9.1 When truthful reporting is uniquely optimal

Strict propriety implies that the only way to achieve the maximum expected score is to announce the exact belief distribution. Whether strictness holds depends on the scoring rule, the outcome space, and the allowed forecast class. For instance, if two distinct beliefs yield the same scores for all outcomes within the model constraints, the rule may not separate them, leading to non-unique optimal reports.

9.2 Flat regions and indifference cases

Some scoring rules produce “flat” expected-score regions where multiple announced distributions yield the same expected score against a given true distribution. This occurs when the rule lacks enough curvature or when the forecast class identifies distributions only up to an equivalence relation. In such cases, incentive compatibility remains, but the reporting behavior may be indeterminate among multiple forecasts.

9.3 Distinguishing different belief distributions

Identifiability refers to the rule’s ability to distinguish belief distributions based on observed outcomes. Strict propriety is one way to ensure identifiability in expectation. In finite samples, however, empirical scores may not fully separate close beliefs, so practical distinguishability depends on sample size, outcome richness, and variance properties of the scoring rule under repeated sampling.

10 Extensions and Generalizations

10.1 Weighted proper scoring rules

Weighted proper scoring rules modify the contribution of different outcomes or regions of the space. By assigning outcome-dependent weights, one can emphasize or de-emphasize certain parts of the distribution. Under suitable conditions, the weighted version remains proper, often because the underlying convex-analytic structure is preserved after reweighting.

10.2 Cost-sensitive variants (overview)

In some applications, errors are not equally harmful across outcomes. Cost-sensitive variants incorporate this by adjusting the scoring function so that mistakes involving high-cost events are penalized more. When constructed carefully, cost-sensitive rules can preserve propriety within the relevant forecast space, ensuring that cost adjustments change the evaluation emphasis without breaking incentive compatibility.

10.3 Proper composite scoring rules (overview)

Composite scoring rules combine simpler scores applied to components of structured predictions. Examples include scoring marginals in a coordinated way, or scoring conditionals within a probabilistic factorization. Proper composite rules aim to maintain global incentive compatibility even though the evaluation is assembled from multiple local scoring functions.

10.4 Mechanism-design and incentive compatibility (light overview)

In mechanism-design settings, proper scoring rules can be embedded into payment schemes that reward participants for reporting probability forecasts. The key requirement is that the expected payment is maximized when truthful beliefs are reported. Under assumptions about participant rationality and the availability of verifiable outcomes, proper scoring rules provide a foundation for incentive-compatible information elicitation and aggregation mechanisms.