1 Loss functions and decision-theoretic background

1.1 Loss vs. score: terminology and relationship

In decision theory, a loss function maps a decision (or prediction) and an outcome to a nonnegative penalty. In forecasting and statistical learning, the same mathematical object is often called a score, especially when smaller values indicate better predictive performance. The distinction is mostly conventional: the core idea is that the model’s output is evaluated by a function that quantifies disagreement with what actually happens.

A frequent viewpoint is to interpret losses through the lens of incentives: if the decision-maker minimizes expected loss, the properties of that loss determine whether truthful probabilistic reporting is rewarded.

1.2 Bayes risk and expected loss

Consider an unknown data-generating mechanism characterized by a distribution \(P\) over outcomes. If a forecaster reports a predictive distribution (or a statistic derived from it) \(\hat P\), the expected loss under \(P\) is an expected loss (or, in Bayesian language, a form of Bayes risk when minimizing over a class of actions). The forecaster’s objective is to choose \(\hat P\) that minimizes this expectation.

The key object is therefore the functional \[ \mathbb{E}_{Y\sim P}\big[L(\hat P, Y)\big], \] and how its minimizers relate to the true distribution \(P\).

1.3 Properness as an incentive property

Properness is an incentive property: a loss is called proper when, for each possible true distribution \(P\), the report that matches \(P\) minimizes expected loss. Under properness, expected-loss minimization leads a rational forecaster to report their true beliefs.

When properness fails, the forecaster may have incentives to report something other than their actual beliefs, even if they attempt to minimize expected loss.

2 Proper and improper losses

2.1 Definition of proper losses

A loss \(L\) is proper if for every true distribution \(P\) in the relevant domain, the expected loss \[ \mathbb{E}_{Y\sim P}[L(Q,Y)] \] is minimized at \(Q=P\). If the minimizer is unique, the loss is strictly proper; otherwise it is proper but not strict.

Properness can be defined for different reporting formats: probability vectors in discrete settings, densities in continuous ones, or other parametric representations. The conceptual requirement stays the same: truthful reporting should be an expected-loss minimizer.

2.2 Definition of improper losses

An improper loss is one that does not satisfy the defining incentive behavior above. In practice, impropriety means there exists at least one distribution \(P\) such that the expected loss is minimized by some reported object \(Q\neq P\), or that minimization does not uniquely point to \(P\) even when \(Q=P\) is among minimizers.

Thus, improperness is about the structure of expected-loss minimizers, not simply about whether the loss is mathematically well-defined.

2.3 Strict vs. non-strict properness

Strict properness is stronger than properness. It rules out the possibility that two different reports yield exactly the same expected loss for a given \(P\). Non-strict properness still encourages truthful reporting but may allow multiple optimal reports.

Improper losses can resemble non-strict proper losses in that multiple minimizers exist; however, the defining difference is that improper losses may not include the truth among the minimizers, or may include it but still fail to deliver the incentive guarantees typically associated with strictness.

2.4 Characterizing improperness: multiple minimizers and reporting incentives

Impropriety can be understood through two related mechanisms:

  1. Multiple minimizers: The expected-loss landscape can have a set of minimizers rather than a single target. This can weaken identifiability of beliefs from optimal actions.
  2. Biased incentives: Even if a model can minimize expected loss, the minimizing report can systematically deviate from the true distribution. In forecasting, this means optimal predictions under the objective do not correspond to calibrated probability forecasts for the underlying uncertainty.

A practical implication is that one can fit an estimator by minimizing an improper objective while still achieving good predictive performance in some sense, yet the learned probabilities may be misaligned with true event frequencies.

3 Examples and canonical improper losses

3.1 Bregman divergences and improper scoring behavior

Many scoring rules can be expressed using Bregman divergences, which are nonnegative quantities generated by a convex function. For a proper scoring rule, the divergence structure typically ensures that the expected divergence is minimized at the truth.

Impropriety arises when the mapping from reports to divergence arguments breaks the required alignment—for example, by applying transformations that change which point plays the role of “report equals truth,” or by using divergences outside the standard probabilistic simplex/density framework without the corresponding normalization conditions.

In effect, the divergence may still measure discrepancy, but the expected-loss minimizer need not coincide with the true distribution.

3.2 Quadratic loss variants (with emphasis on calibration issues)

Quadratic loss is often presented as a convenient error measure. In simple regression, minimizing squared error can be linked to conditional means. However, when the task is probabilistic forecasting, using quadratic-type losses on probability parameters or transformed quantities can create calibration problems.

For instance, if a forecaster is asked to report probabilities but the objective corresponds to minimizing squared error on a nonlinear transform (or on a proxy not equal to the probability), the optimal report can be systematically different from the true probability. The forecaster may minimize error in the transformed space while producing biased probabilities in the original space.

3.3 Log-likelihood and common transforms (where impropriety can arise)

The negative log-likelihood is a canonical scoring rule for density forecasts when used correctly. It is closely tied to likelihood-based estimation and, under suitable conditions, yields proper scoring behavior.

Impropriety can appear when the log score is altered in ways that destroy the connection between expected score and entropy-like quantities. Common sources include:

  • applying incorrect normalization,
  • using improper link functions between the model output and the probability scale,
  • scoring after a non-invertible transform that changes the effective prediction target.

The result is an objective that may still correlate with predictive fit but whose minimizer does not necessarily correspond to true beliefs over outcomes.

3.4 Weighted and truncated objectives as sources of impropriety

In many applications, practitioners use modifications to improve robustness or computation. Weighted losses and truncated objectives can change the balance between regions of the outcome space. For example:

  • weighting some outcomes more heavily than others can shift the expected-loss minimizer,
  • truncating tails can reduce sensitivity to extreme events but also breaks the exact correspondence that proper scoring rules rely upon.

These alterations are not inherently problematic; they may be beneficial for robustness or operational priorities. However, they can introduce impropriety by altering which report minimizes expected loss under the modified objective.

4 Consequences in forecasting and estimation

4.1 Calibration implications

A central consequence of improper loss is calibration failure. Calibration asks whether predicted probabilities match observed frequencies. With proper losses, minimizing expected loss typically aligns predictions with true frequencies. With improper losses, the optimization can drive the model toward predictions that are optimal for the chosen objective yet systematically miscalibrated relative to the underlying event probabilities.

This can manifest as overconfident or underconfident forecasts, especially when impropriety is strong or when the domain of allowable reports interacts with the loss’s minimizer set.

4.2 Incentive and identifiability effects

Improper objectives can also create issues of incentive and identifiability. If multiple reports minimize the expected loss, then observational data may not uniquely determine which beliefs produced the optimal predictions. Even worse, if the truth is not among the minimizers, then the optimal model may converge to a surrogate of the truth rather than an identifiable belief.

In communication settings, this corresponds to a forecaster having no guarantee that rationally minimizing the objective yields truthful beliefs.

4.3 Estimation bias under improper objectives

When an estimator is derived by minimizing an improper objective, the resulting estimator can be biased as an estimator of the “true” probabilistic target (e.g., the conditional probability of an event). Importantly, the bias here is not only a finite-sample effect; it can be population-level—the limiting estimator under infinite data still aims at a different target induced by the objective.

This phenomenon is especially visible when the improper loss induces a projection onto a mismatched parameterization or a transformed target.

4.4 Robustness vs. correctness trade-offs

Improper losses are sometimes used intentionally to obtain desirable computational or statistical properties. For example, truncation and weighting can make learning more stable under outliers or heavy-tailed noise. From a decision-theoretic standpoint, this reflects a trade-off: robustness and optimization convenience may come at the cost of losing correctness in the sense of calibrated probabilistic reporting.

In practice, the question becomes whether the application prioritizes predictive ranking, operational utility, or truthful probability calibration. Impropriety is one lever that can move along that trade-off curve.

5 Learning and optimization perspective

5.1 Surrogate objectives and why improper losses appear

Many learning algorithms rely on surrogate objectives: tractable losses used in place of an intractable or non-differentiable target. Surrogates are not required to be proper. Improperness often appears because the surrogate is designed to optimize some proxy metric (e.g., margin separation, robust deviation, or computational tractability) rather than to enforce truthful scoring incentives.

A loss can therefore be improper even when it is carefully derived and empirically useful.

5.2 Consistency of estimators under improper loss

Consistency asks whether an estimator converges to the desired parameter as sample size grows. Under improper losses, one can still have consistency with respect to an alternative target: the estimator converges to the population minimizer of the objective, which may correspond to a transformed notion of the truth.

Thus, the right concept is not always “consistency for the true probability,” but consistency for the objective-induced target.

5.3 Regularization, constraints, and induced impropriety

In real models, the loss is rarely used alone. Regularization (penalties on parameters) and constraints (limited model capacity, restricted output domains, monotonicity requirements) can effectively alter the decision problem. Even if the base loss is proper in an unconstrained space, the constraints can prevent reaching the true minimizer, leading to behavior that resembles impropriety at the effective prediction layer.

This “induced impropriety” is common in constrained probabilistic modeling and in parameterizations that cannot represent the full set of beliefs.

5.4 Practical training considerations (local minima, scaling, gradients)

Optimization landscapes under improper losses can differ from those under proper ones. The gradient signal may correspond to improving expected objective value, but that does not guarantee movement toward calibrated beliefs. Additional considerations include:

  • scaling or normalization of the loss across batches,
  • numerical transformations (e.g., logits to probabilities),
  • gradient behavior that can favor certain regions of the prediction space.

These factors can amplify the incentive mismatch produced by the objective.

6 Measuring and diagnosing impropriety

6.1 Checking whether truthful reporting is optimal

A direct diagnostic is to ask whether the true distribution (or the correct conditional belief) minimizes the expected objective. In practice, this is often done indirectly, using theoretical derivations for the loss form and the assumed family of distributions.

When the model class is restricted, the test becomes more subtle: one must compare the induced population minimizer within the class, not the unrestricted truth.

6.2 Comparing predicted vs. realized frequencies

Calibration-focused diagnostics provide an empirical window into impropriety. If predicted probabilities are systematically too high or too low, the mismatch can indicate that the loss’s incentive does not align with truthful probabilistic reporting.

Common tools include reliability plots and binning-based comparisons, though these evaluate calibration rather than establishing properness itself.

6.3 Uncertainty calibration tests

Beyond frequency matching, one can test whether forecast distributions match observed uncertainty patterns. In regression settings, this corresponds to checking whether prediction intervals achieve nominal coverage under repeated sampling. In classification settings, it includes assessing whether uncertainty estimates correspond to actual error rates.

Poor outcomes can be symptomatic of improper scoring or of model misspecification interacting with an improper objective.

6.4 Sensitivity analysis to objective choice

Another diagnostic approach is to train (or analyze) using multiple candidate objectives and compare which one yields calibration and decision performance aligned with the intended interpretation. If changing the loss alters the meaning of “best prediction” in a way that harms calibration, the current objective may be improper relative to the desired reporting target.

This method is practical because it treats improperness as an empirical property: the “correct” objective is the one whose optimization produces behavior consistent with the intended semantics.

7 Connecting improper loss to information theory

7.1 Divergences, entropy, and expected score interpretations

Proper scoring rules often connect to entropy and divergences between distributions. Improper losses can break or distort this relationship. When the expected score no longer corresponds to an entropy-like functional, the objective may fail to privilege truthful reports.

Even when a loss resembles an information-theoretic divergence, small modifications—such as transforms, weights, or domain restrictions—can shift the expected-score minimizer away from the distribution used to define the divergence.

Improper losses also relate to regret: the difference in expected loss between the chosen report and the best achievable report under the objective. When the objective is improper, the “best achievable report” can differ from the agent’s true beliefs, so regret minimization does not imply belief recovery or calibration.

This helps explain why two agents with different beliefs might both minimize the same objective in ways that do not reflect their true uncertainty.

7.3 Mutual information–style viewpoints (intuition)

Information-theoretic intuitions can be transferred to improper scoring by considering how objectives reward reduction of uncertainty. Proper losses often quantify uncertainty in a way that aligns with truthful beliefs, resembling an information gain concept. Improper losses, by contrast, may quantify a different criterion—one that does not correspond cleanly to information about the true latent distribution.

While the analogy is not always literal, it provides intuition for why expected-loss minimization can lead away from calibrated probability reporting.

8 Special cases and extensions

8.1 Multiclass and structured prediction settings

In multiclass classification, losses such as cross-entropy are commonly proper when used as standard log scores over probability vectors. Improperness can arise when the training objective differs from the intended probability score, for example through:

  • scoring margins instead of probabilities,
  • using non-probabilistic targets or constrained outputs,
  • applying weights that correspond to changing the effective data-generating distribution.

In structured prediction, where outputs have complex dependencies, improper surrogates are frequent because exact proper scoring over the full combinatorial space can be computationally infeasible.

8.2 Continuous outcomes and density estimation

For continuous variables, proper density scoring requires carefully defined objects like densities and normalization. Improperness can occur when the model outputs do not correspond to a true density (e.g., due to missing normalization constants) or when the scoring rule uses approximate likelihoods, discretizations, or surrogate error measures that do not match the density semantics.

These issues can lead to objectives that optimize goodness-of-fit in a transformed space rather than minimizing a divergence with the correct minimizer.

8.3 Distributional losses vs. pointwise losses

Some objectives score entire distributions; others evaluate only point predictions or pointwise residuals. A loss designed for pointwise accuracy may be improper as a distributional score if the mapping from distribution to point forecast is not aligned with the probability that the report is meant to communicate.

Conversely, a distributional loss may reduce to a pointwise measure under special parameterizations; the induced properness depends on that parameterization.

8.4 Handling missing data and censoring with improper objectives

When data are incomplete or subject to censoring, practitioners often use specialized objectives to accommodate the missingness mechanism. Improperness can enter if the objective assumes an approximation to the likelihood or uses heuristic weighting that changes which predictive distribution minimizes expected loss.

If the missingness model is correct and the objective corresponds to a proper likelihood-based score, calibration may be preserved. Otherwise, incentive misalignment can lead to biased uncertainty estimates.

9 Remedies and alternatives

9.1 Converting an improper loss to a proper one

A common remedy is to re-express the objective in a form that is known to be proper for the reporting target. In some settings, one can find a transformation or correction that maps the improper scoring rule to a proper scoring rule, restoring truthful incentives.

This often requires identifying the functional form of the objective and constructing an associated scoring rule whose expected value is minimized at the true distribution.

9.2 Composite objectives and correction terms

Another approach uses composite objectives: combining an improper component with a correction term designed to adjust incentives. In statistical learning, such corrections can be interpreted as compensating for a mismatch between the training loss and the target decision criterion.

The feasibility depends on whether the correction can be computed or estimated reliably and whether the correction preserves desirable optimization properties.

9.3 Choosing loss functions for the intended reporting target

Because impropriety is relative to the semantic meaning of the prediction, the most robust remedy is choosing a loss aligned with the desired target—probabilistic calibration, ranking, interval coverage, or robustness. If the application requires calibrated probabilities, proper scoring rules on the probability scale are often appropriate. If the goal is robustness or ranking, a different criterion may be acceptable, but then the interpretation of outputs should be adjusted accordingly.

This reframing avoids treating impropriety as a flaw when the objective was never meant to support truthful probabilistic reporting.

9.4 When keeping impropriety is beneficial

Improproper losses can be beneficial when the evaluation metric differs from probabilistic calibration. For example, objectives tailored to operational costs or decision thresholds may deliberately distort probability semantics to improve downstream utility.

In such cases, the key requirement is transparency: users should understand that the model’s outputs optimize a surrogate criterion and may not be valid as calibrated probabilities without additional post-processing.

10 Summary and key takeaways

10.1 Checklist for interpreting improper loss in practice

  • Identify the reporting target: Are outputs intended to be calibrated probabilities, point estimates, or decision-oriented scores?
  • Check properness relative to that target: Determine whether the expected objective is minimized at the truth.
  • Look for calibration failure: Reliability plots, coverage checks, and frequency alignment can indicate incentive mismatch.
  • Distinguish population behavior from finite-sample issues: Improperness can create asymptotic bias toward an objective-induced target.
  • Consider constraints and regularization: These can induce incentive deviations even with a base proper loss.

10.2 Common pitfalls and how to avoid them

A frequent pitfall is assuming that a “good” predictive objective automatically yields meaningful probabilities. Improper losses can produce accurate predictions while still misrepresenting uncertainty. Another common error is applying likelihood-like intuition to transformed or weighted versions of a loss without verifying whether the transformation preserves proper scoring behavior.

To avoid these issues, one should match the loss to the intended semantics, validate with calibration diagnostics when probabilities are required, and apply correction methods or choose proper alternatives when truthful reporting is a design goal.