1 Background and Motivation
1.1 Why compare probability distributions
Comparing probability distributions is central to statistics, machine learning, and probability theory. Many tasks—such as assessing whether two datasets come from the same source or measuring how learning algorithms change a model—require a quantitative notion of “distance” between distributions. Classical choices include differences in means, variances, or pointwise evaluations, but these can miss important aspects of distributional shape, such as spread, multimodality, or geometric relationships among values.
Wasserstein distances address this limitation by comparing distributions through the geometry of the underlying space. Instead of treating probability mass as fixed at points, they allow mass to be transported from one distribution to another, producing a distance that reflects both where mass lies and how it must shift to match.
1.2 Earth mover’s distance intuition
A common intuition is the “earth mover’s distance” (also used as a synonym for \(W_1\) in some contexts). Imagine one distribution as a pile of earth on a location set and another as a different pile. Moving a unit of earth from position \(x\) to position \(y\) costs an amount that depends on the distance between \(x\) and \(y\). The Wasserstein distance is the minimum total cost required to transform the first pile into the second, assuming the piles have equal total mass.
This viewpoint makes Wasserstein distances sensitive to the cost of moving probability mass, not merely to whether two distributions have similar numerical summaries.
1.3 Connections to coupling and optimal transport
Wasserstein distances belong to optimal transport theory. The key concept is a coupling (also called a transport plan), which specifies how mass at points in one distribution pairs with mass at points in another. Among all couplings that respect the marginal distributions, the optimal transport plan minimizes the expected transport cost.
In this framework, the Wasserstein metric can be interpreted as the cost of best possible coordinated “matching” between random variables drawn from the two distributions, when the cost of pairing outcomes depends on their separation in the space.
2 Mathematical Definition
2.1 Notation and probability measures
Let \((\mathcal{X}, d)\) be a metric space with distance function \(d\). Let \(\mu\) and \(\nu\) be probability measures on \(\mathcal{X}\). Fix an order \(p \ge 1\). The \(p\)-Wasserstein distance will quantify how expensive it is to transport \(\mu\) into \(\nu\) when moving mass over distance \(d(x,y)\) costs \(d(x,y)^p\).
2.2 Transport plans and couplings
2.2.1 Feasible marginals
A transport plan is a probability measure \(\pi\) on \(\mathcal{X}\times\mathcal{X}\) whose marginals match \(\mu\) and \(\nu\). Concretely, \(\pi\) is feasible if for measurable sets \(A,B\subseteq \mathcal{X}\), \[ \pi(A\times \mathcal{X})=\mu(A),\qquad \pi(\mathcal{X}\times B)=\nu(B). \] The collection of all feasible couplings is denoted by \(\Pi(\mu,\nu)\).
2.2.2 Cost function and transport cost
Given a feasible coupling \(\pi\), the expected transport cost under \(\pi\) is \[ \int_{\mathcal{X}\times\mathcal{X}} d(x,y)^p\,\mathrm{d}\pi(x,y). \] The Wasserstein distance is obtained by minimizing this quantity over all feasible couplings.
2.3 \(W_p\) definition (general order)
For \(p\ge 1\), the \(p\)-Wasserstein distance between \(\mu\) and \(\nu\) is \[ W_p(\mu,\nu) \;=\; \left(\inf_{\pi\in\Pi(\mu,\nu)} \int_{\mathcal{X}\times\mathcal{X}} d(x,y)^p\,\mathrm{d}\pi(x,y)\right)^{1/p}, \] with the convention that the infimum may be infinite if no finite-cost transport exists.
2.4 Special case: \(W_1\) (first Wasserstein distance)
For \(p=1\), \[ W_1(\mu,\nu) \;=\; \inf_{\pi\in\Pi(\mu,\nu)} \int d(x,y)\,\mathrm{d}\pi(x,y). \] This case is particularly important because it admits strong dual characterizations and often yields meaningful geometric behavior even when only first moments are controlled.
2.5 Special case: \(W_2\) (second Wasserstein distance)
For \(p=2\), \[ W_2(\mu,\nu) \;=\; \left(\inf_{\pi\in\Pi(\mu,\nu)} \int d(x,y)^2\,\mathrm{d}\pi(x,y)\right)^{1/2}. \] Squared cost makes \(W_2\) especially natural in settings connected to geometry, least-squares objectives, and certain dynamical interpretations of optimal transport.
3 Equivalent Formulations
3.1 Kantorovich formulation (minimization over transport plans)
The definition above is the Kantorovich formulation: it optimizes over all couplings \(\pi\). This relaxation is powerful because it allows splitting mass, not requiring a deterministic transport rule. As a result, the problem is convex in \(\pi\) (under appropriate assumptions), which supports existence results and computational methods.
3.2 Monge formulation (minimization over transport maps)
The Monge formulation seeks a measurable map \(T:\mathcal{X}\to\mathcal{X}\) pushing \(\mu\) forward to \(\nu\), denoted \(T_\#\mu=\nu\). The goal is to minimize \[ \int d(x,T(x))^p\,\mathrm{d}\mu(x). \] This formulation restricts transport to be deterministic: each point \(x\) sends all its mass to a single destination \(T(x)\). In general spaces and for general measures, such a map may not exist, which is why the Monge problem can be harder than the Kantorovich relaxation.
3.3 Duality theory
Duality provides alternative ways to compute or interpret Wasserstein distances. Dual forms replace the minimization over transport plans by a maximization over potential functions, revealing links to functional inequalities, continuity, and convergence.
3.3.1 Kantorovich–Rubinstein duality for \(W_1\)
For \(W_1\) on \(\mathcal{X}\), under suitable conditions, the Kantorovich–Rubinstein duality states that \[ W_1(\mu,\nu) \;=\; \sup_{f}\left\{\int f\,\mathrm{d}\mu-\int f\,\mathrm{d}\nu\right\}, \]
| where the supremum is over all 1-Lipschitz functions \(f\) (i.e., \( | f(x)-f(y) | \le d(x,y)\)). This duality turns the transportation problem into a problem about distinguishing distributions using Lipschitz test functions. |
|---|
3.3.2 Dual potentials and interpretability
In dual formulations, optimal potentials can be viewed as “witnesses” of disagreement between \(\mu\) and \(\nu\). For instance, a maximizing 1-Lipschitz function highlights regions where the distributions differ while respecting the geometry of the metric space. Although these potentials may not be unique, their existence and properties help explain why \(W_1\) is stable under certain perturbations and why it relates to notions of smooth discrepancy.
3.4 Relation to cumulative distributions in 1D
In one dimension, the geometry simplifies considerably and \(W_p\) can be expressed using distribution functions or quantiles.
3.4.1 Quantile function representation
Let \(F_\mu^{-1}\) and \(F_\nu^{-1}\) denote quantile functions. For \(p\ge 1\), when \(\mathcal{X}\subseteq\mathbb{R}\), \[
| W_p(\mu,\nu)^p = \int_0^1 \big | F_\mu^{-1}(u)-F_\nu^{-1}(u)\big | ^p\,\mathrm{d}u. |
|---|
\] This formula shows that, in 1D, the optimal coupling matches quantiles at the same probability level.
3.4.2 Closed-form expressions in simple cases
For special families—such as distributions with non-overlapping support under simple geometries—closed-form expressions may be derived from quantile relations. For discrete measures on a line, one can compute \(W_p\) by tracking cumulative mass and matching it along the line, which often reduces to piecewise calculations.
4 Existence, Finiteness, and Regularity
4.1 Conditions for finite \(W_p\)
The Wasserstein distance \(W_p(\mu,\nu)\) is finite if the measures have enough moment control relative to the metric. A standard condition is that both \(\mu\) and \(\nu\) have finite \(p\)-th moments (with respect to a reference point), ensuring that transporting far-out mass does not incur infinite cost. On general metric spaces, finiteness criteria depend on the growth of the cost function \(d(x,y)^p\).
4.2 Moment requirements
Moment requirements are tied to the integrability of transport cost. Roughly, if either distribution places too much probability mass at arbitrarily large distances from a fixed origin (in \(\mathbb{R}^n\), from the origin), then any coupling must transport some mass over large distances, potentially making the cost infinite.
4.3 Metric properties and topology induced by \(W_p\)
When \(W_p(\mu,\nu)<\infty\), the Wasserstein distance defines a metric on the set of measures with finite \(p\)-th moment. In terms of topology, convergence in \(W_p\) strengthens weak convergence by requiring additional moment alignment. Thus, Wasserstein topology captures both distributional shape and tail behavior.
4.4 Relationship to weak convergence
4.4.1 Moment-constrained convergence
A typical principle is that \(W_p(\mu_n,\mu)\to 0\) implies \(\mu_n\) converges weakly to \(\mu\), and the convergence rate is influenced by how moments behave. Conversely, weak convergence plus uniform control of \(p\)-th moments can imply convergence in \(W_q\) for \(q\le p\). This interplay is central in probability theory and statistical asymptotics using Wasserstein distances.
5 Computation and Algorithms
5.1 Discrete distributions and linear programming
5.1.1 Transport matrix formulation
When \(\mu\) and \(\nu\) are discrete with supports \(\{x_i\}_{i=1}^m\) and \(\{y_j\}_{j=1}^n\), the problem becomes a finite linear program. A transport plan can be represented by a nonnegative matrix \(\gamma\in\mathbb{R}^{m\times n}\) satisfying marginal constraints: \[ \sum_{j=1}^n \gamma_{ij} = a_i,\qquad \sum_{i=1}^m \gamma_{ij} = b_j, \] where \(a_i\) and \(b_j\) are masses of \(\mu\) and \(\nu\). The cost is \(\sum_{i,j} \gamma_{ij} d(x_i,y_j)^p\), and the infimum is over feasible \(\gamma\).
5.1.2 Complexity considerations
The naive linear-programming approach can be computationally expensive because the number of variables grows as \(mn\). For large-scale problems, specialized algorithms, sparsity exploitation, and approximations are often required. Complexity also depends on the chosen solver and the structure of the cost matrix (e.g., whether it admits fast evaluation or has special forms).
5.2 Entropic regularization (Sinkhorn approach)
5.2.1 Regularized objective
To obtain tractable computation, entropic regularization adds a penalty term proportional to the entropy of the transport plan. For discrete measures, one solves \[ \min_{\gamma\in\Pi(a,b)} \sum_{i,j}\gamma_{ij}C_{ij} + \varepsilon\sum_{i,j}\gamma_{ij}(\log \gamma_{ij}-1), \] where \(C_{ij}=d(x_i,y_j)^p\) and \(\varepsilon>0\) controls the strength of regularization. This yields a unique and smooth solution and makes optimization easier.
5.2.2 Iterative normalization steps
The Sinkhorn algorithm solves the regularized problem via iterative row/column rescaling. Each iteration alternates between enforcing the row sums and column sums to match the desired marginals. The method is widely used because it converts a global constrained optimization into efficient matrix operations.
5.3 Stochastic / minibatch approximations
5.3.1 Estimators for empirical measures
In statistical practice, \(\mu\) and \(\nu\) are often observed through samples, leading to empirical measures \(\hat{\mu}\) and \(\hat{\nu}\). Wasserstein distances between empirical measures can be computed approximately using subsampling, minibatching, or stochastic optimization strategies. Such methods aim to balance computational cost with estimator accuracy.
5.4 Practical issues: numerical stability and scaling
Implementations must address numerical overflow or underflow, particularly in entropically regularized settings where \(\gamma_{ij}\) may become extremely small. Scaling the cost matrix, choosing \(\varepsilon\) appropriately, and using stable log-domain computations are common techniques. Additionally, for high-dimensional data, the cost of evaluating pairwise distances and storing large cost matrices becomes a dominant bottleneck.
6 Properties and Theoretical Results
6.1 Bounds and comparisons with other distances
6.1.1 Total variation vs Wasserstein
Total variation measures the maximum discrepancy of probabilities over measurable sets, but it is insensitive to the geometry of the space. Wasserstein distances, by contrast, account for how far probability mass must move, so they can be smaller than total variation when distributions are close in a transportation sense. Comparisons between the two depend on assumptions about metric structure and moment bounds.
6.1.2 MMD and Wasserstein relationships (high level)
Maximum Mean Discrepancy (MMD) compares distributions using reproducing kernel Hilbert space features, which can also reflect shape differences. While neither Wasserstein nor MMD universally dominates the other, there are connections under regularity conditions: each can be interpreted as measuring discrepancies induced by test functions, and their effectiveness can depend on the choice of kernel (for MMD) and the order \(p\) (for Wasserstein).
6.2 Lipschitz and continuity properties
Wasserstein distances behave well under perturbations of measures. In particular, if a sequence of measures converges and appropriate moments remain controlled, then \(W_p\) distances to the limit vanish. Dual representations provide continuity estimates with respect to Lipschitz test functions, supporting stability claims in both theoretical and numerical studies.
6.3 Triangle inequality and metric behavior
As a metric (on measures with finite \(p\)-th moments), \(W_p\) satisfies the triangle inequality: \[ W_p(\mu,\eta)\le W_p(\mu,\nu)+W_p(\nu,\eta). \] This property is useful in chaining bounds across intermediate distributions, including in approximation schemes and error analyses.
6.4 Concentration and empirical Wasserstein distance
6.4.1 Rates in terms of sample size and dimension
The empirical Wasserstein distance \(W_p(\hat{\mu}_n,\mu)\) depends on sample size and ambient dimension. In many settings, convergence can degrade with increasing dimension, reflecting the “curse of dimensionality.” The precise rates vary with \(p\), the dimension, and the regularity of \(\mu\), but the overall phenomenon is that more samples are required to accurately estimate transport-based distances in high-dimensional spaces.
7 Applications in Probability and Statistics
7.1 Convergence in distribution using Wasserstein metrics
7.1 Tightness and moment control
Wasserstein metrics strengthen convergence in distribution by incorporating quantitative information about tails. In proofs, tightness and moment estimates are often used to show that transport costs do not blow up. This makes Wasserstein convergence a natural tool when studying limiting behavior of random variables, especially when distributions shift while keeping some moment structure.
7.2 Stochastic processes and dynamical systems
In stochastic analysis, Wasserstein distances can measure how probability laws evolve over time. They appear in stability estimates for Markov processes and in comparisons between solutions of stochastic or deterministic dynamical systems when randomness is present in initial conditions or forcing terms.
7.3 Robustness and distributional perturbations
Because Wasserstein distances quantify minimal transport effort, they serve as a notion of robustness: small Wasserstein distance between a nominal distribution and a perturbed one implies that downstream quantities sensitive to distributional changes vary in a controlled way, often under Lipschitz assumptions on the quantities of interest.
7.4 Model evaluation in distribution learning (overview)
In generative modeling, one frequently evaluates how well a learned distribution matches a target distribution. Wasserstein metrics are used to compare generated samples to real data distributions, sometimes with regularization to enable computation. Their geometric sensitivity makes them informative when generated samples are in the correct regions but may miss fine details or mode structure.
8 Wasserstein Distance Variants
8.1 Conditional and sliced Wasserstein distances
Conditional versions assess distance between conditional distributions given covariates. Sliced Wasserstein distances reduce computation by projecting distributions onto one-dimensional subspaces and averaging the resulting 1D Wasserstein distances. These variants can improve scalability while retaining some geometric interpretability.
8.2 Wasserstein barycenters
A Wasserstein barycenter is a distribution that minimizes the weighted sum of Wasserstein distances to a collection of distributions. In applications such as averaging images or combining probability models, barycenters provide a principled way to interpolate between distributions while respecting geometry.
8.3 Wasserstein GAN objective (conceptual overview)
In generative adversarial networks, a Wasserstein-based objective replaces classical divergence measures with Wasserstein distance (or a regularized proxy). The discriminator is interpreted through a function constrained by Lipschitz regularity, and training aims to minimize the distance between the generated distribution and the data distribution. Conceptually, it frames learning as transporting probability mass from the generated distribution toward the data distribution.
8.4 Regularized and generalized transport costs
Beyond entropic regularization, other generalizations modify the cost function or impose constraints that reflect the application. Examples include different power costs, alternative metrics, or penalties that encourage certain structures in transport plans. These extensions can address modeling needs or improve computational behavior.
9 Worked Examples
9.1 Comparing two 1D distributions via quantiles
Consider two distributions on \(\mathbb{R}\) with cumulative distribution functions \(F_\mu\) and \(F_\nu\). For \(p=1\), the 1D representation gives \[
| W_1(\mu,\nu) = \int_0^1 \big | F_\mu^{-1}(u)-F_\nu^{-1}(u)\big | \,\mathrm{d}u. |
|---|
\] To compute it, one evaluates the quantile functions over the probability levels \(u\in[0,1]\) and integrates the absolute difference. This procedure directly reflects how far mass must be shifted at each quantile level.
9.2 Example with discrete measures (small support)
Let \(\mu\) place masses \(a_1,a_2\) at points \(x_1,x_2\) and \(\nu\) place masses \(b_1,b_2\) at points \(y_1,y_2\). The coupling matrix \(\gamma\) has four entries \(\gamma_{ij}\) with constraints matching row sums to \((a_1,a_2)\) and column sums to \((b_1,b_2)\). With \(W_1\), the cost becomes a weighted sum of distances: \[
| \sum_{i=1}^2\sum_{j=1}^2 \gamma_{ij}\, | x_i-y_j | . |
|---|
\] For small supports, one can often solve the feasible constraints and choose the plan minimizing the cost, sometimes resulting in a simple “move along the line” interpretation.
9.3 Example illustrating \(W_1\) dual characterization
Suppose \(\mu\) and \(\nu\) are distributions on \(\mathbb{R}\). By the Kantorovich–Rubinstein duality, \(W_1(\mu,\nu)\) equals the supremum over all 1-Lipschitz \(f\) of \(\int f\,\mathrm{d}\mu-\int f\,\mathrm{d}\nu\). In practice, one may consider candidate functions \(f\) that emphasize regions where \(\mu\) has more mass than \(\nu\). As the candidate class becomes richer (while preserving the Lipschitz constraint), the computed discrepancy approaches \(W_1(\mu,\nu)\). This illustrates that Wasserstein distance can be understood as the strongest separation achievable by smoothly varying test functions.