1 Introduction to Proper Loss

1.1 Motivation from probabilistic prediction

In probabilistic forecasting, a model typically outputs a full predictive distribution rather than a single point estimate. Proper loss functions are designed so that a forecaster is rewarded (or penalized) in a way that makes reporting the correct predictive distribution the best strategy. This property is important for both statistical evaluation of forecasts and for training predictive models, because it aligns the optimization target with accurate uncertainty quantification.

1.2 Proper loss vs. improper loss

A loss is called *proper* (more precisely, a *proper scoring rule* when expressed in terms of predictive distributions) if the forecaster minimizes the *expected* loss by choosing the forecast distribution that matches the data-generating or evaluation distribution. If a loss is *improper*, there can be systematic incentives to misreport probabilities, so optimization may produce forecasts that are biased toward the loss function rather than toward calibration with respect to the true distribution.

1.3 Relation to scoring rules

Proper losses are closely related to scoring rules: functions that assign a numerical score (or, equivalently, a loss) after observing outcomes. Many results in forecasting and statistics are phrased in the language of scoring rules, where “proper” indicates that truthful reporting of probabilities maximizes expected score. In machine learning and statistics, loss-based training is often implemented directly with proper loss functions such as log loss or the Brier score.

2 Mathematical Foundations

2.1 Expected loss and risk

Let \(Y\) denote an outcome and let \(p\) be a distribution from which \(Y\) is drawn. A loss function \(\ell(q, y)\) takes a reported distribution \(q\) and an observed outcome \(y\). The expected loss (also called risk) under \(p\) is \[ L(p, q)=\mathbb{E}_{Y\sim p}[\ell(q,Y)]. \] Properness concerns how \(L(p,q)\) behaves as a function of \(q\) when \(p\) is fixed.

2.1.1 Conditioning and evaluation distributions

In practice, the evaluation distribution may be conditional on available information \(X\). One typically considers a conditional risk \[ L(p(\cdot\mid X), q(\cdot\mid X)), \] and averages it over the distribution of \(X\). Properness then ensures that, for each information state, the expected penalty is minimized by reporting the conditional distribution that governs the outcomes given that state.

2.2 Optimality and strictness

A loss is *proper* if for every admissible true distribution \(p\), the minimum of \(L(p,q)\) over all admissible reports occurs at \(q=p\). It is *strictly proper* if this minimizer is unique (up to equivalence induced by the function’s domain), so that any deviation from \(p\) increases expected loss. Strictness prevents ambiguity and is central for identifiability of the intended target distribution.

2.3 Properness and identifiability

Properness ties the loss minimizer to the target distribution. When a loss is strictly proper, minimizing expected loss uniquely identifies the distribution \(p\). Without strictness, multiple reports may yield the same expected risk, which can still be useful but may not uniquely pin down the distribution or latent parameters used to generate it.

3 Proper Loss for Discrete Outcomes

3.1 Categorical predictions

For discrete outcomes, consider a finite set \(\{1,\dots,K\}\). A forecaster reports a probability vector \(q=(q_1,\dots,q_K)\) in the simplex. After observing outcome \(y\in\{1,\dots,K\}\), the loss \(\ell(q,y)\) specifies the penalty. Properness ensures that the expected loss under the true distribution \(p\) is minimized by reporting \(q=p\).

3.2 Characterizing proper losses via Bayes acts

A standard approach characterizes proper losses through *Bayes acts*: actions (here, reported distributions) that minimize expected loss for a given belief \(p\). Properness means that the Bayes act for belief \(p\) is to report \(p\) itself. This viewpoint generalizes easily and yields constructive ways to design losses by specifying which reports are Bayes acts under different beliefs.

3.3 Examples of proper loss functions

3.1 Log loss and cross-entropy

For categorical data, the log score uses \[ \ell(q,y)=-\log q_y \] (with \(q_y>0\)). If \(p\) is the true distribution, minimizing the expected log loss yields \(q=p\). The expected log loss can be written in terms of the negative entropy of \(p\) plus a divergence term, so that the minimum occurs when the divergence is zero.

3.2 Brier score (quadratic loss)

Another canonical example is the Brier score: \[ \ell(q,y)=\sum_{i=1}^K (q_i-\mathbf{1}\{y=i\})^2. \] Its expected value under \(p\) is minimized at \(q=p\). In binary settings this reduces to squared error of the probability estimate. Quadratic loss is widely used because of its interpretability and favorable statistical properties, though it behaves differently from log loss in terms of tail sensitivity.

3.3 Ranked and transformed probability scores

Beyond basic choices, one can design proper scores using transformations of probabilities or using structure that depends on ordering or grouping of classes. Such scores can maintain properness while adjusting how disagreement in probability mass is penalized across different outcomes. Transformations typically arise from linking proper scores to convex function generators or divergence-like quantities.

4 Proper Loss for Continuous Predictions

4.1 Forecasting with densities

For continuous outcomes, a forecaster may report a probability density \(q(y)\) over a space \(\mathcal{Y}\). A proper loss assigns a penalty based on the density value at the realized observation or based on comparisons between the predicted distribution and the observation in an integral form. The goal remains the same: minimize expected loss under the true distribution \(p\) by choosing \(q=p\).

4.2 Proper loss under integrability conditions

Continuous proper scoring rules require conditions so that expectations exist and values are well-defined (e.g., integrability of certain functions of densities). Under suitable regularity, a scoring rule is proper if its expected value is minimized at the truth. Many results mirror discrete ones but require more careful handling of measurability and absolute continuity.

4.2.1 Proper scoring rules for predictive distributions

When forecasts are distributions rather than merely parameter vectors, proper scoring rules can be described in terms of how expected scores relate to divergence measures or convex functionals over the space of distributions. This approach supports constructing losses for a wide range of model classes, including parametric families where \(q\) is represented by a parameter.

4.3 Common continuous examples

4.3.1 Log score for densities

The continuous analog of log loss assigns \[ \ell(q,y)=-\log q(y), \] interpreted in the density sense. Under appropriate conditions (such as positivity of \(q(y)\) where \(p\) has support), minimizing expected log score again recovers \(q=p\). The log score is closely tied to maximum likelihood estimation because maximizing log score corresponds to maximizing likelihood when the predictive distribution is parameterized.

4.3.2 Energy-based and divergence-linked scores

Some continuous proper losses are formulated using integral functionals resembling “energy” between distributions. Others are linked to divergence measures that quantify mismatch between \(p\) and \(q\). In these settings, properness is obtained by choosing a score whose expected value decomposes into a constant term plus a nonnegative discrepancy that vanishes only at equality of distributions.

5 Strict Properness and Uniqueness

5.1 When proper losses are strictly proper

Strict properness holds when the only way to achieve the minimal expected loss is to report the true distribution exactly (again, modulo any equivalences induced by the scoring rule’s structure). In common examples, strictness depends on how the loss reacts to deviations across the support of the distribution and on whether the function penalizes all forms of mismatch.

5.2 Failure modes and non-unique optima

A loss may be proper but not strictly proper if different reports yield the same expected risk, for example when scores are insensitive to certain aspects of the predictive distribution or when admissible reports are restricted in a way that collapses distinctions. Non-unique minimizers can still yield calibrated predictions under some objectives, but they complicate interpretation because the data do not distinguish between multiple reported distributions.

5.3 Sensitivity to misspecification

When the forecaster’s model class cannot represent the true distribution, minimizing expected proper loss leads to an approximation that is optimal within the constraint set rather than equal to the truth. The mismatch between the model class and the target affects calibration and the meaning of the “best” report. Understanding strictness helps clarify whether the optimization landscape has a unique best approximation or may offer multiple competing forecasts with similar risk.

6.1 Bregman divergences connection

Many proper losses can be related to *Bregman divergences*, which arise from convex functions. In such cases, the expected loss under the true distribution decomposes into a constant plus a Bregman divergence between the true and predicted quantities. This explains why the truth minimizes expected loss and why strictness corresponds to strict convexity of the generator.

6.2 f-divergences connection

Another family of relationships connects expected proper scores to *\(f\)-divergences*. The precise mapping depends on the chosen scoring rule and the way densities are compared. These links provide both intuition and tools: divergence nonnegativity yields properness, and divergence equality conditions yield strictness when the divergence vanishes only at matching distributions.

6.3 Calibration interpretations

Calibration describes whether predicted probabilities match observed frequencies or conditional distributions. Proper losses provide incentive-compatible mechanisms for achieving calibration: if a forecaster were to repeatedly report probabilities derived from their current beliefs and optimize expected proper loss, the reports are driven toward calibrated behavior. In evaluation contexts, minimizing proper loss is therefore not just accuracy of point estimates but a rigorous way to assess probabilistic alignment.

6.4 Proper loss as an incentive for truthful reporting

From an incentive perspective, properness means that truthful reporting is a dominant or optimal strategy in expectation. This can be framed as an elicitation problem: the designer chooses a scoring mechanism so that participants have no advantage in misrepresenting their subjective probabilities. While practical settings involve finite samples and model constraints, the theoretical backbone of proper losses is that they reward honest belief states.

7 Constructing Proper Loss Functions

7.1 General construction principles

Constructing proper losses often begins with specifying how one wants the score to respond to differences between distributions. A common principle is to build a scoring rule whose expected value is minimized when a convex discrepancy measure is zero. This can be implemented via convex analysis, functional derivatives, or by specifying Bayes acts and working backward to obtain a loss that has those acts as minimizers.

7.2 Using convex functions and subgradients

Convex function methods are central because many scoring rules correspond to gradients or subgradients of convex generators. This yields a systematic route from a convex potential to an elicitation function (or scoring rule). When the generator is strictly convex, one typically obtains strict properness, and the geometry of the convex function determines the shape of the penalty.

7.3 Elicitation functions and representation results

7.3.1 Characterizations for binary outcomes

For binary outcomes, proper scoring rules admit simpler representations. One can describe how the expected score depends on the predicted probability and derive conditions for properness directly. Such results often express scores through derivatives of a convex function, with strictness tied to strict convexity and boundary behavior.

7.3.2 Characterizations for general outcome spaces

For larger discrete spaces and continuous settings, representation results generalize the binary case using convex functionals over the probability simplex or space of densities. While the details depend on topology, measurability, and admissibility constraints, the overarching idea remains: a proper scoring rule can be constructed from a convex generator, and expected risk corresponds to a constant minus a divergence-like quantity.

8 Computational and Practical Considerations

8.1 Estimation under empirical risk minimization

In practice, proper losses are optimized using empirical risk minimization, replacing the population expectation with an average over observed samples. The incentive property motivating properness is then translated into consistency: with sufficient data and appropriate model capacity, minimizing empirical proper loss tends toward the distribution that minimizes expected risk within the chosen model class.

8.2 Numerical stability (e.g., log loss)

Some proper losses, especially log-based ones, can be numerically unstable when predicted probabilities become extremely small or zero at observed outcomes. Implementations often use smoothing, clipping, or constrained optimization to prevent undefined logarithms and to control gradient magnitudes. These safeguards preserve the practical usefulness of proper losses while approximating their theoretical behavior.

8.3 Regularization and constraints on predictions

Modeling constraints—such as restricting outputs to a parametric family, enforcing probability simplex constraints, or applying regularization penalties—affect how closely the optimization follows the ideal proper-loss objective. Regularization can improve generalization but may also introduce bias relative to the distribution that would minimize pure expected loss. Understanding these interactions is key for interpreting trained probabilistic forecasts.

8.4 Handling missing or censored outcomes

When data are incomplete or outcomes are censored, direct application of a scoring rule may not be straightforward because the realized outcome is not fully observed. Practical adaptations include using conditional scoring under an observation model, integrating over missing variables, or employing likelihood-consistent methods. Properness can be preserved when these adaptations correspond to the correct conditional expectation of the intended proper score.

9 Applications and Use Cases

9.1 Forecast evaluation and benchmarking

Proper losses provide a principled way to compare forecasting systems. Because they reward well-calibrated probability assignments, comparing average proper loss across models yields an assessment that reflects both sharpness and calibration. In benchmark settings, these metrics help distinguish systems that are accurate in frequency but miscalibrated, from those that achieve a better overall probabilistic match.

9.2 Training probabilistic machine learning models

Many probabilistic models—such as classifiers that output class probabilities and density models that represent continuous uncertainty—are trained using proper losses like cross-entropy or log score. The training objective aligns with maximum likelihood under common model parameterizations, and it encourages calibrated probability outputs when model capacity and data assumptions are reasonable.

9.3 Aggregating expert opinions

Proper loss functions support combining forecasts from multiple experts. When each expert provides a predictive distribution, one can use proper scoring mechanisms to aggregate information in a way that incentives consistent reporting and yields an aggregate forecast that performs well in expected loss. Such aggregation can be viewed as an application of Bayesian-like updating or decision-theoretic pooling.

9.4 Online learning with proper losses

In online learning, forecasters update predictions sequentially based on incoming data. Proper losses fit naturally because they can yield regret bounds and convergence guarantees under suitable assumptions. The incentive-compatible property helps ensure that the learning dynamics are directed toward minimizing true expected risk rather than exploiting quirks of an improper objective.

10.1 Multivariate and structured prediction

Structured prediction involves predicting complex objects (sequences, graphs, or other structured outputs) where dependencies matter. Proper losses extend to these settings by defining scoring rules over conditional distributions on structured spaces. Properness then corresponds to correct elicitation of the entire predictive distribution, not merely marginal probabilities.

10.2 Distributional robustness and worst-case risk

Robustness methods consider uncertainty about the true distribution. Extensions of proper loss frameworks may optimize worst-case expected scores over sets of plausible distributions. This leads to objectives that balance calibration with protection against distribution shift, while maintaining an elicitation interpretation within the chosen uncertainty set.

10.3 Consistency and regret bounds

Proper loss-based learning can be analyzed in terms of consistency (convergence of predictions to risk minimizers) and regret (performance relative to a benchmark strategy). The mathematical connection between properness and divergence-like quantities often makes it easier to derive rates and to interpret what it means for a learning algorithm to improve probabilistic predictions.

10.4 Connection to maximum likelihood estimation

The log loss is intimately connected to maximum likelihood estimation. When a model class defines conditional densities \(q_\theta\), minimizing expected log loss corresponds to maximizing expected log likelihood. As a result, maximum likelihood can be interpreted as optimizing a proper loss objective, with theoretical implications for calibration and for the behavior of predicted probabilities under correct model specification.