1 Discrepancy function basics

1.1 Definition and core properties

A discrepancy function is a mapping that assigns a nonnegative value to a comparison between an object produced by a model, predictor, or approximation procedure and a corresponding target or ideal quantity. The central idea is to turn “difference” into a single number that can be minimized, monitored, or compared across competing methods. In many settings, the compared objects can be outcomes (e.g., predicted vs. observed values), functions (e.g., two curves), probability distributions (e.g., empirical vs. theoretical), or sets of samples.

Although discrepancy functions vary widely in form, they often share a common intent: smaller values indicate closer agreement, while larger values quantify increasing disagreement.

1.2 Relationship to distance and error measures

Discrepancy functions are closely related to error measures, and in many cases they behave like distances. However, not every discrepancy is a metric. Some are tailored to specific tasks—such as emphasizing particular regions of the input space, tolerating certain errors more than others, or reflecting practical costs of deviations. As a result, a discrepancy function may satisfy only parts of metric structure (for instance, it may be nonnegative but fail symmetry).

1.3 Interpretation of scale and units

A key aspect of using discrepancy functions is understanding what the numerical scale means. Depending on the construction, the value can have direct physical or statistical interpretation (e.g., squared units of a quantity) or a dimensionless form (e.g., normalized errors). Weighted or integrated discrepancies may also depend on domain size or the choice of weighting scheme, so the magnitude is not always transferable between problems without careful normalization.

1.4 Common axioms (nonnegativity, identity, symmetry)

Many discrepancy functions enforce nonnegativity. An often-desired “identity” property states that the discrepancy is zero when the approximation matches the target (under the chosen notion of equality). Symmetry—swapping arguments does not change the value—is common for distance-like constructions, but it may not hold for loss-oriented or direction-sensitive comparisons.

Even when these properties do not hold exactly, designers frequently aim for behavior that is intuitive for interpretation: consistent penalization of mismatch and stability under minor perturbations.

2 Mathematical formulations

2.1 Pointwise discrepancy

2.1.1 Absolute vs squared discrepancies

Pointwise discrepancy compares two values at a location, typically an observed quantity and a predicted one. Common choices include the absolute difference \(y-\hat{y}\) and the squared difference \((y-\hat{y})^2\). Absolute discrepancies grow linearly with the magnitude of error and are more robust to outliers. Squared discrepancies grow quadratically and thus penalize large deviations more aggressively.

When aggregated over many points, these pointwise forms often become mean absolute error or mean squared error, respectively.

2.1.2 Residuals and loss functions

A residual is the raw difference between a target and a prediction, often written as \(r = y-\hat{y}\) or \(r=\hat{y}-y\). Pointwise discrepancy can be expressed as a function of residuals, such as \(\phi(r)=r\) or \(\phi(r)=r^2\). In learning and estimation, such \(\phi\)-based discrepancies are tightly connected to loss functions: the discrepancy represents how strongly deviations should be penalized during fitting.

2.2 Integral discrepancy

2.2.1 Weighted integrals over domains

For function-valued targets (e.g., approximating a curve), discrepancy may be defined as an integral of pointwise differences over a domain \(\Omega\). A typical pattern is \[ D(f,g)=\int_\Omega w(x)\,\phi\big(f(x)-g(x)\big)\,dx, \] where \(w(x)\) is a weight that controls which regions matter more. This framework is useful when errors are not equally important everywhere, or when the domain has varying reliability or relevance.

2.2.2 Kernel-based discrepancy forms

Kernel methods express discrepancies using similarity structures. In many cases, a discrepancy can be written through kernels that emphasize certain features or scales in the data. Kernel-based discrepancy forms are particularly common when comparisons are made in high-dimensional feature spaces without explicitly computing those features. Conceptually, they transform the task from “difference in raw coordinates” to “difference in learned or chosen representations.”

2.3 Distributional discrepancies

2.3.1 Empirical vs theoretical distribution comparison

A distributional discrepancy compares a distribution represented by samples or an empirical measure to a target distribution, or compares two empirical distributions. Typical inputs include probability densities, cumulative distribution functions, or sample sets that define empirical measures. Such discrepancies quantify how far the modeled or sampled behavior deviates from expected probabilistic structure.

2.3.2 Divergences and distance-like metrics

In distributional contexts, discrepancies include divergences (often direction-sensitive) and distance-like metrics (symmetric). Divergences can arise from information theory or from variational formulations and may depend on how probabilities are transformed. Distance-like metrics often satisfy additional structural properties that make them easier to interpret as “movement” between distributions, though they may come with different computational costs.

2.4 Function-space discrepancy

2.4.1 Norms and semi-norms

For comparing functions directly, discrepancy can be defined using norms in a function space. For instance, an \(L^p\) norm of the difference \(f-g\) yields a family of discrepancies that weight errors differently across magnitude. Norms that satisfy the full norm axioms provide strong geometric meaning; semi-norms relax some requirements and may treat distinct functions as equivalent under the chosen measure of difference.

2.4.2 Reproducing kernel Hilbert space approaches

Reproducing kernel Hilbert spaces (RKHS) provide a flexible setting for discrepancy definitions. By measuring the size of the difference between functions in an RKHS, one obtains discrepancies that reflect similarity under the kernel’s notion of smoothness or feature alignment. RKHS-based discrepancies are widely used because they can be estimated from samples and can connect to statistical hypothesis testing and distribution comparison.

3 Key examples in practice

3.1 Mean absolute error and mean squared error

Mean absolute error aggregates absolute residuals across samples: \[

\text{MAE}=\frac{1}{n}\sum_{i=1}^ny_i-\hat{y}_i.

\] Mean squared error aggregates squared residuals: \[ \text{MSE}=\frac{1}{n}\sum_{i=1}^n (y_i-\hat{y}_i)^2. \] MAE tends to be more tolerant of occasional large errors, while MSE strongly emphasizes them. Both are widely used because of their simplicity and interpretability.

3.2 L1/L2 norm discrepancies

In vector-valued settings, discrepancy can be defined as an \(L^1\) or \(L^2\) norm of the error vector \(e=\hat{\theta}-\theta\):

- \(L^1\): \(\|e\|_1 = \sum_je_j\)
- \(L^2\): \(\|e\|_2 = \sqrt{\sum_j e_j^2}\)

Choosing between these norms affects the resulting optimization geometry. \(L^1\) can encourage sparsity in certain parameter-estimation frameworks, whereas \(L^2\) is closely tied to least-squares formulations and Gaussian error models.

Total variation provides a distributional discrepancy that reflects the largest possible difference between probabilities assigned by two measures. It can be interpreted as a worst-case discrepancy over events and is closely connected to coupling and classification-type viewpoints. In practice, estimating it may be challenging in high dimensions, but it offers a strong conceptual benchmark for “how distinguishable” two distributions are.

3.4 Wasserstein-type discrepancies (conceptual overview)

Wasserstein-type discrepancies measure how costly it is to transport probability mass from one distribution to another under a specified ground distance. Conceptually, they incorporate the geometry of the space where the distributions live, yielding sensitivity to not just probability differences but also where those differences occur. Such discrepancies can be more informative than purely overlap-based comparisons, especially when one distribution is a shifted or smoothly transformed version of another.

3.5 Discrepancy in numerical integration

3.5.1 Star-discrepancy and uniformity

In numerical integration using deterministic points (e.g., low-discrepancy sequences), star-discrepancy quantifies deviation from uniform coverage of a domain such as \([0,1]^d\). It evaluates how much the empirical point distribution differs from ideal uniformity on axis-aligned subregions anchored at the origin. Lower discrepancy corresponds to more even coverage and improved integration performance for many integrands.

3.5.2 Discrepancy for quasi–Monte Carlo sampling

Quasi–Monte Carlo (QMC) methods replace random sampling with carefully constructed deterministic sequences. Discrepancy measures act as predictors of how well these sequences will approximate integrals: better uniformity typically reduces integration error. The relationship is not universal, but discrepancy-based diagnostics help compare sampling constructions and justify design choices.

4 Construction and selection

4.1 Choosing a target: what is being matched

The “target” in a discrepancy function defines what the approximation must resemble. It could be a ground-truth value, an empirical reference, a prior distribution, or a desired functional relationship. In practice, the choice determines the meaningfulness of the discrepancy: a mismatch between what is being compared and what matters for the downstream objective can yield misleading conclusions.

4.2 Choosing weights and metrics

Weights and underlying metrics shape how errors are penalized. A weight function can prioritize certain regions (e.g., high-confidence or high-cost areas), while a metric can encode geometry (e.g., scaling differences between features). These choices often reflect domain knowledge and can significantly affect optimization behavior and interpretability.

For distributional comparisons, the choice of divergence or distance-like measure effectively selects which aspects of distributional difference are emphasized—such as tail behavior, mean alignment, or full shape matching.

4.3 Robustness to outliers

Discrepancy functions differ in their sensitivity to extreme deviations. Absolute-based discrepancies generally provide more robustness than squared ones in the presence of outliers. For heavy-tailed noise or sporadic measurement errors, robust discrepancy families can prevent a small number of points from dominating the fit.

4.4 Sensitivity analysis across discrepancy choices

Different discrepancies can lead to different parameter estimates and different conclusions about model quality. A practical approach is sensitivity analysis: evaluate how conclusions change when using alternative discrepancy functions (or altered weights). Large variation suggests that the problem may be underdetermined under the chosen notion of mismatch, or that the data contain structures better captured by another measure.

4.5 Calibration and normalization strategies

To compare discrepancies across datasets, models, or scales, normalization is often necessary. Common strategies include dividing by a characteristic magnitude, using scale-invariant formulations, or transforming outputs so that error units become comparable. Calibration may also involve adjusting discrepancy to align with probabilistic assumptions or known noise levels, improving the correspondence between discrepancy magnitude and expected performance.

5 Optimization and experimental design

5.1 Minimizing discrepancy as an objective

Many algorithms treat discrepancy minimization as their core objective. In supervised learning, model parameters are chosen to reduce an error discrepancy between predictions and observed targets. In simulation-based modeling, parameters can be tuned so that the simulated output distribution closely matches empirical data under a chosen discrepancy.

Minimization turns “how different” into a concrete search criterion. The practical challenge is ensuring that the discrepancy is computationally tractable, differentiable when needed, and aligned with the real performance goals.

5.2 Parameter estimation from discrepancy criteria

Parameter estimation can proceed by minimizing discrepancy between model outputs and observations or by matching summary statistics. When the discrepancy is differentiable and the model is smooth, gradient-based methods can be employed. When the discrepancy is expensive to evaluate or non-smooth, derivative-free or surrogate-based optimization may be used.

The statistical meaning of the resulting estimator depends on the discrepancy choice and the data-generating assumptions; different discrepancies can yield estimators with different bias-variance trade-offs.

5.3 Optimal design principles using discrepancy

5.3.1 Space-filling and uniform sampling intuition

Experimental design often aims to gather information efficiently. Discrepancy measures can guide selection of design points to ensure coverage of the input space. The intuition is that well-distributed samples reduce uncertainty and improve the representativeness of model fits, which corresponds to smaller discrepancy in uniformity-based diagnostics.

5.3.2 Sequential design updates

Sequential design repeatedly chooses new experiment settings based on current model uncertainty or current discrepancy behavior. After each batch of observations, the discrepancy landscape is updated, and the next points are selected to reduce expected mismatch. This adaptive strategy can be more sample-efficient than one-shot design, especially when evaluations are costly.

5.4 Stopping rules and convergence criteria

Stopping criteria can depend on the magnitude of discrepancy, its rate of decrease, or stability across iterations. Convergence may be assessed by monitoring changes between successive discrepancy values or by checking whether further optimization yields diminishing returns relative to computation cost. In stochastic settings, confidence intervals or averaged discrepancy trends are often used.

6 Statistical interpretation

6.1 Connection to likelihood and risk (high level)

At a high level, discrepancies relate to risk and uncertainty in statistical modeling. Under certain assumptions, minimizing a discrepancy corresponds to maximizing a likelihood or minimizing an expected loss. For example, squared error is connected to Gaussian noise models, while absolute error aligns with Laplace-like noise assumptions. While not all discrepancy functions have direct likelihood interpretations, many can be tied to probabilistic objectives.

6.2 Generalization and overfitting considerations

A discrepancy computed on training data can be small even when a model performs poorly on new inputs. Generalization depends on how the discrepancy interacts with model complexity, regularization, and data quantity. Overfitting risks are especially relevant when discrepancy definitions are too flexible or when weights emphasize noise rather than signal.

6.3 Asymptotic behavior and consistency intuition

As sample size grows, some discrepancy-based procedures become consistent, meaning they converge toward the true target under suitable conditions. The exact conditions depend on identifiability, the model class, and the discrepancy’s ability to distinguish the correct target from alternatives. Even when consistency holds, the speed of convergence can vary substantially across discrepancy choices.

6.4 Bootstrap and uncertainty assessment (conceptual)

Uncertainty assessment can be performed by resampling strategies such as bootstrap. The general idea is to repeatedly approximate the discrepancy on resampled datasets, producing an empirical distribution of discrepancy values. This helps quantify variability and can support comparisons between models or hyperparameter settings when analytical uncertainty is difficult to derive.

7 Computation and approximation methods

7.1 Analytical evaluation vs numerical estimation

Some discrepancy functions admit closed-form evaluation, especially for simple models or special choices such as norms on finite-dimensional vectors. Others require numerical integration or optimization over auxiliary variables. When direct evaluation is unavailable, approximation methods are used, trading accuracy for computational feasibility.

7.2 Monte Carlo estimation of discrepancy

Monte Carlo approaches estimate discrepancies by sampling from relevant distributions. For integral discrepancies, random sampling can approximate integrals. For distributional discrepancies, sampling can approximate expectations or variational bounds. The accuracy depends on variance and sample size, and careful estimator design can reduce computational burden.

7.3 Empirical plug-in estimators

In many tasks, the discrepancy is defined in terms of unknown quantities (e.g., true distributions). Empirical plug-in estimators replace unknown elements with estimates derived from data, such as empirical measures or fitted densities. Plug-in methods are popular due to simplicity, though they can introduce bias or additional variance, particularly in small-sample regimes.

7.4 Computational complexity considerations

Computational cost depends on how discrepancy is evaluated and optimized. Complexity can grow with dimensionality, sample size, or kernel complexity. For example, kernel-based discrepancies may require operations that scale with the number of samples unless approximations are used. For distributional comparisons, optimization-based discrepancies like transport distances can be particularly demanding without efficient approximations.

8 Applications and use cases

8.1 Model validation and goodness-of-fit

Discrepancy functions are used to assess whether a model’s outputs align with observed data. In goodness-of-fit workflows, discrepancy provides a single number summarizing mismatch, enabling comparison across models. When calibrated appropriately, discrepancy statistics can also support diagnostic interpretation—such as identifying whether a failure is due to systematic bias or localized errors.

8.2 Simulation studies and benchmark comparisons

In simulation-based research, discrepancies quantify how closely an algorithm’s output matches a known reference. Benchmarking often relies on discrepancy functions to provide consistent evaluation across conditions. The choice of discrepancy can affect what a benchmark measures (e.g., mean accuracy vs. distribution shape), so careful alignment with goals is essential.

8.3 Surrogate modeling and surrogate error tracking

Surrogate models approximate expensive simulators. Discrepancy functions can monitor how well the surrogate reproduces simulator outputs and guide adaptive refinement by identifying where the surrogate is inaccurate. Using discrepancies that reflect the surrogate’s intended use can improve practical reliability.

8.4 Signal processing and regression residual assessment

In signal processing, residual-based discrepancies help assess model fit and detect deviations such as noise levels, artifacts, or model mismatch. Regression and filtering workflows frequently employ discrepancy summaries to compare reconstructed signals against targets, assess the quality of parameter estimates, or trigger re-estimation when performance degrades.

9 Limitations and pitfalls

9.1 Misleading discrepancy scales

A discrepancy value can be numerically small yet still represent poor practical performance if the discrepancy scale does not match task relevance. Conversely, a large discrepancy might be acceptable if it occurs in regions that have little impact on the ultimate objective. Interpreting magnitudes therefore requires linking discrepancy to real outcomes and units.

9.2 Non-identifiability under certain discrepancies

Some discrepancy functions may fail to distinguish between different underlying targets. For instance, if a discrepancy ignores certain aspects of the data representation, multiple models can achieve the same discrepancy value while differing in unmeasured ways. This can lead to non-identifiability: optimization cannot uniquely recover the intended target.

9.3 Dependence on representation and preprocessing

Discrepancies depend on how inputs and outputs are represented. Feature scaling, transformations, normalization, and choice of basis can substantially change discrepancy behavior. As a result, two pipelines that are equivalent under the hood may yield different discrepancy values after preprocessing changes, complicating comparison and interpretation.

9.4 When discrepancy does not correlate with task performance

Discrepancy measures can correlate weakly with downstream performance if the discrepancy does not capture what matters operationally. This mismatch can occur when the discrepancy is used as a proxy for a different objective (e.g., optimizing distribution shape when the end task depends primarily on decision boundaries). Validation should confirm that reduced discrepancy corresponds to improved utility.

10.1 Loss functions vs discrepancy functions

Loss functions are often used during training and may be defined per data instance or expectation over data. Discrepancy functions are broader: they can compare approximations to targets in many forms, including distributions and function spaces. In many practical settings, a loss function can be a specific type of discrepancy, but the terms are not always interchangeable.

10.2 Divergences and metrics

Divergences and metrics both quantify differences, with divergence often allowing asymmetry and sometimes failing the triangle inequality. Metrics usually satisfy symmetric and triangle-inequality properties. Discrepancy functions include both kinds, depending on the chosen mathematical structure.

10.3 Residual-based diagnostics

Residuals and their aggregates provide diagnostic insight into where a model fails. Residual-based discrepancies can reveal systematic patterns such as heteroscedasticity, nonlinearity, or temporal drift. Diagnostics often combine discrepancy values with residual plots or additional summary statistics.

10.4 Uniformity measures and sampling quality metrics

Uniformity measures, including discrepancy measures for deterministic point sets, evaluate how evenly samples cover a region. These metrics are central to quasi–Monte Carlo methods and to experimental design strategies that seek space-filling coverage. Their role is to connect sampling geometry to approximation quality.