1 Introduction to distribution comparison
1.1 Two-sample problems and distributional difference
Comparing two probability distributions is a central task in statistics and machine learning. In a two-sample setting, one observes two independent samples drawn from unknown distributions and seeks to quantify how different the underlying distributions are. The goal may be descriptive (measuring discrepancy), predictive (aligning distributions so a model generalizes better), or inferential (testing whether the samples come from the same source).
1.2 Kernel methods and RKHS intuition
Kernel methods compare data through similarities measured by a kernel function. A kernel implicitly maps inputs into a high-dimensional feature space where linear operations can represent nonlinear relationships in the original space. The reproducing kernel Hilbert space (RKHS) formalizes this idea: probability distributions can be represented by their embedded means in the RKHS, enabling a geometric notion of distance between distributions.
1.3 Mean embeddings of probability distributions
The key idea behind maximum mean discrepancy is that a probability distribution can be summarized in RKHS by its mean embedding. If a distribution is pushed through the kernel feature map, the average feature vector characterizes the distribution. Comparing two distributions then becomes comparing these average embeddings. When the embedding map is rich enough, the distance between embeddings reflects the actual difference between the distributions.
2 Definition of Maximum Mean Discrepancy
2.1 Population MMD in RKHS
Let \(X\) and \(Y\) be random variables with distributions \(P\) and \(Q\). Choose a positive-definite kernel \(k(\cdot,\cdot)\) with RKHS \(\mathcal{H}\) and feature map \(\phi\). The population MMD is defined as the RKHS norm of the difference between mean embeddings: \[
| \mathrm{MMD}(P,Q)=\left\|\mathbb{E}_{X\sim P}[\phi(X)]-\mathbb{E}_{Y\sim Q}[\phi(Y)]\right\|_{\mathcal{H}}. |
|---|
\] This quantity measures how far the embedded “average” feature representations of the two distributions are from each other.
2.2 Kernel choice and characteristic kernels
The kernel determines which aspects of the distributions are compared. Some kernels guarantee that \(\mathrm{MMD}(P,Q)=0\) implies \(P=Q\); such kernels are called characteristic. With characteristic kernels, MMD behaves like a proper discrepancy measure rather than collapsing distinct distributions into the same embedding. In practice, kernel properties and scaling can materially affect sensitivity.
2.3 Properties as a divergence-like metric
MMD is nonnegative and symmetric by construction. Whether it satisfies the triangle inequality depends on the specific setting, so it is often described as a “divergence-like” distance rather than a strict metric in all cases. Still, it provides a practical notion of discrepancy with a clear interpretation: the discrepancy is the maximal difference in expectations over functions drawn from the unit ball of the RKHS.
2.4 Relationship to other distribution distances
MMD relates to several other measures. It is connected to integral probability metrics, where distances are defined as supremums of differences in expectations across function classes. It also connects to likelihood-free comparisons in spirit, while differing from metrics such as KL divergence or total variation. Unlike some divergences, MMD does not require density estimation in standard formulations and can be computed directly from samples.
3 Empirical estimation of MMD
3.1 Biased (V-statistic) estimator
Given samples \(\{x_i\}_{i=1}^n \sim P\) and \(\{y_j\}_{j=1}^m \sim Q\), an empirical estimate of MMD can be computed by expanding the squared RKHS norm: \[ \widehat{\mathrm{MMD}}^2 = \frac{1}{n^2}\sum_{i=1}^n\sum_{i'=1}^n k(x_i,x_{i'}) + \frac{1}{m^2}\sum_{j=1}^m\sum_{j'=1}^m k(y_j,y_{j'}) - \frac{2}{nm}\sum_{i=1}^n\sum_{j=1}^m k(x_i,y_j). \] This corresponds to a biased estimator because it includes “diagonal” terms where indices coincide.
3.2 Unbiased (U-statistic) estimator
An alternative estimator removes diagonal terms to reduce bias: \[ \widehat{\mathrm{MMD}}^2_{\text{unb}} = \frac{1}{n(n-1)}\sum_{i\neq i'} k(x_i,x_{i'}) + \frac{1}{m(m-1)}\sum_{j\neq j'} k(y_j,y_{j'}) - \frac{2}{nm}\sum_{i=1}^n\sum_{j=1}^m k(x_i,y_j). \] This U-statistic form is often preferred in hypothesis testing because its expectation matches the population quantity under suitable conditions.
3.3 Computing MMD from samples
Computation reduces to evaluating kernel values among samples and aggregating them according to the chosen estimator. Many implementations leverage Gram matrices \(K_{xx}\), \(K_{yy}\), and \(K_{xy}\) to avoid repeated kernel calls. The squared form is commonly used for optimization and testing because it is differentiable when kernels are smooth and because it avoids square roots.
3.4 Computational complexity and scaling
The naive cost of forming all pairwise kernel evaluations is \(O(n^2+m^2+nm)\), which can become expensive for large sample sizes. Scaling strategies include using mini-batches, approximating kernels via random features, employing low-rank Gram matrix approximations, or using linear-time unbiased estimators constructed from random pairings. The choice depends on whether the priority is statistical accuracy or computational efficiency.
4 MMD in hypothesis testing
4.1 Two-sample test setup
MMD can be used to test the null hypothesis \(H_0: P=Q\) against alternatives where the distributions differ. Under \(H_0\), the expected MMD value is near zero, while under the alternative, it tends to be larger. The test statistic is typically an empirical estimate of squared MMD.
4.2 Permutation testing and p-values
A common approach is permutation testing: combine both samples, randomly split them into two groups of sizes \(n\) and \(m\), compute the test statistic for each permutation, and compare the observed statistic to the permutation distribution. This yields a p-value that does not require strong assumptions about the underlying distributions, making it robust in practice.
4.3 Asymptotic null distributions
For large samples, theoretical results characterize the distribution of the MMD-based statistic under \(H_0\). The limiting distribution may involve weighted sums of chi-squared random variables (depending on the kernel and estimator choice). Such results allow approximate p-values and facilitate analytic power and calibration considerations when permutation testing is too costly.
4.4 Multiple-testing considerations
In workflows where many pairs of distributions or many candidate models are tested, controlling false discoveries becomes important. Standard multiple-testing strategies (e.g., family-wise error or false discovery rate control) can be applied to MMD-based tests. Additionally, careful handling of dependence between tests may be needed if overlapping samples are reused across comparisons.
5 Kernel design and practical considerations
5.1 Common kernels (RBF, linear, polynomial)
| Typical choices include the linear kernel \(k(x,x')=x^\top x'\), polynomial kernels, and the radial basis function (RBF) kernel \(k(x,x')=\exp(-\|x-x'\|^2/(2\sigma^2))\). Each kernel emphasizes different structure: linear kernels focus on mean differences in feature space, while RBF kernels can capture more complex nonlinear discrepancies depending on bandwidth. Polynomial kernels balance between these behaviors. |
|---|
5.2 Bandwidth selection for RBF kernels
For RBF kernels, the bandwidth \(\sigma\) strongly influences sensitivity. Very small \(\sigma\) can make the kernel overly localized, causing MMD to reflect noise rather than systematic shifts. Very large \(\sigma\) can overly smooth the kernel, reducing contrast between distributions. Heuristics such as median distance rules, cross-validation driven by downstream objectives, or multi-kernel mixtures are often used.
5.3 Feature sensitivity and over/under-smoothing
Kernel hyperparameters effectively determine the feature resolution. Under-smoothing (too localized) may yield unstable estimates and high variance, while over-smoothing (too broad) may under-detect differences, leading to low power. The effect propagates into both testing and training-based discrepancy minimization. Monitoring behavior across a grid of parameters can help diagnose whether the kernel is appropriate.
5.4 Normalization and stability
MMD computations can be sensitive to scaling of inputs and numerical precision. Normalizing features (e.g., standardization) often improves kernel stability, especially for distance-based kernels. For training scenarios, practitioners sometimes use normalized MMD variants or gradient clipping to avoid exploding gradients when batch sizes are small or kernel values saturate.
6 MMD for domain adaptation and transfer learning
6.1 Distribution alignment objective
Domain adaptation seeks models that generalize from a source domain to a target domain whose data distribution differs. MMD provides an alignment criterion that penalizes discrepancies between the feature distributions induced by a model. The basic idea is to map inputs to a representation space where source and target samples become more similar in the RKHS sense.
6.2 End-to-end training with MMD loss
In deep learning, a feature extractor produces representations \(f(x)\), and MMD is computed between source and target representations. The training objective includes a task loss (e.g., classification loss on labeled source data) plus an MMD penalty encouraging alignment. Gradients backpropagate through the feature extractor, allowing end-to-end optimization of both predictive performance and distribution matching.
6.3 Conditional vs. marginal MMD
Marginal MMD aligns overall feature distributions without conditioning on labels. Conditional MMD aligns distributions within each class by computing discrepancy between source and target features among corresponding predicted or true labels. Conditional alignment can be more effective when label distributions differ or when class-conditional shifts dominate, but it may require careful handling of label information and prediction noise.
6.4 Combining MMD with task-specific losses
MMD is typically combined with supervised or self-supervised objectives to maintain task-relevant structure. Overemphasizing alignment can lead to representations that match across domains but lose discriminative information. Balancing weights between discrepancy penalties and task losses is therefore critical, often achieved through validation-based tuning or scheduling strategies during training.
7 Variants and extensions
7.1 Conditional MMD (CMMD)
Conditional MMD extends the alignment principle by measuring discrepancy between conditional distributions \(P(\phi(X)\mid Y)\) and \(Q(\phi(X)\mid Y)\) in RKHS. In practice, CMMD is estimated using samples grouped by labels (or pseudo-labels) and aggregated across classes. This can improve robustness to situations where marginal alignment alone would mix distinct classes.
7.2 Class-conditional and stratified MMD
A related family of approaches computes MMD separately per class and then combines the results, often with weights reflecting class frequencies. Stratified estimators can reduce variance and improve interpretability, because they reveal which classes contribute most to distribution mismatch. They also help when the domains differ primarily in certain categories.
7.3 Multi-kernel MMD
Rather than committing to a single kernel, multi-kernel MMD combines several kernels with different bandwidths or functional forms. This can yield more stable behavior across unknown scales in the data. Weighting can be fixed heuristically or learned, enabling the discrepancy measure to adapt to multiple modes of variation.
7.4 Wasserstein-inspired and other kernelized extensions
MMD is one member of a broader class of kernelized discrepancy measures. Extensions inspired by optimal transport concepts attempt to incorporate transport-like behavior while retaining tractable kernel computations. Other variants modify function classes or use alternative regularization to capture more nuanced distribution differences. The guiding theme remains: represent distributions in a feature space and compute discrepancy through a principled distance or IPM.
8 Theoretical perspectives
8.1 Consistency and identifiability
Theoretical guarantees for MMD depend heavily on kernel properties. With characteristic kernels, MMD-based discrepancy is identifiable: zero population discrepancy implies equal distributions. Consistency results establish that empirical MMD estimates converge to the population value as sample sizes grow, under assumptions about kernel boundedness and sample independence.
8.2 Convergence rates
Convergence rates describe how quickly empirical MMD approaches its population counterpart. Rates differ between biased and unbiased estimators and depend on the smoothness and boundedness of the kernel, as well as the underlying distribution. These results inform practical decisions about sample size and estimator choice when the objective is accurate discrepancy estimation or reliable hypothesis testing.
8.3 Sample-efficiency and variance behavior
MMD’s statistical efficiency depends on how much information the kernel captures about differences between distributions. If the kernel is well matched to the structure of the discrepancy, estimates can be accurate with fewer samples. Poor kernel choices can increase estimator variance and reduce power. Batch-based approximations also introduce variability that can be characterized to guide trade-offs between computation and accuracy.
8.4 Generalization interpretation via RKHS norms
When MMD is used as a training penalty, theoretical interpretations often connect it to generalization bounds. Informally, aligning distributions in feature space can reduce the difference in expected errors between domains. In RKHS terms, the penalty corresponds to a norm-based measure of how differently functions in the hypothesis class behave across domains, linking discrepancy control to generalization performance.
9 Practical workflow and diagnostics
9.1 Choosing an estimator (biased vs. unbiased)
For optimization tasks, the biased squared estimator is commonly used because it is simple and numerically convenient. For formal two-sample testing, unbiased estimators may be advantageous for calibration. The choice can also depend on sample sizes: when one group is small, unbiased estimators may have higher variance due to reduced effective sample pairs.
9.2 Choosing a kernel and tuning parameters
Kernel selection is typically treated as a hyperparameter decision. Practitioners often start with an RBF kernel due to flexibility, tune bandwidth via heuristics or validation, and consider polynomial or linear kernels when features are already well behaved. Multi-kernel strategies can reduce brittleness when the appropriate scale is unclear.
9.3 Monitoring training curves with MMD
In domain adaptation settings, MMD values can be tracked over training iterations alongside task metrics. A decreasing MMD curve may indicate alignment progress, but it does not guarantee improved predictive performance. Conversely, MMD may stagnate if the model cannot represent domain-invariant features or if optimization dynamics conflict with task objectives.
9.4 Interpreting results and failure modes
Large MMD values can indicate significant distribution shift, but they may also arise from mismatch in preprocessing, feature scaling, or kernel sensitivity. When MMD-driven training fails, common causes include poorly tuned kernel bandwidth, insufficient model capacity, incorrect weighting between MMD and task losses, or class-imbalance issues affecting conditional alignment. Diagnosing these requires combining discrepancy measures with validation performance and qualitative checks.
10 Summary and further reading
10.1 Key takeaways
Maximum mean discrepancy offers a principled way to quantify and minimize differences between probability distributions using RKHS mean embeddings. Its behavior depends strongly on kernel choice, but with characteristic kernels it provides an identifiable discrepancy. Empirical estimates can be computed efficiently for moderate sample sizes, and MMD can be used for both hypothesis testing and representation learning, especially in domain adaptation.
10.2 Suggested references and benchmark resources
For foundational theory and practical guidance, standard references include texts on kernel methods and statistical learning, as well as seminal papers introducing MMD and its use in two-sample testing and domain adaptation. Benchmark resources often accompany MMD-related methods in machine learning libraries and research papers that report results on domain adaptation datasets, distribution shift benchmarks, and kernel tuning ablations.