1 Background and Motivation
1.1 Relative entropy and the role of KL divergence
| Relative entropy, commonly denoted Kullback–Leibler (KL) divergence, is a directed measure of how one probability distribution differs from another. For distributions \(P\) and \(Q\) on the same space, \(\mathrm{KL}(P\|Q)\) quantifies the expected log-likelihood ratio under \(P\). In many statistical and learning settings, KL divergence arises naturally from likelihood-based reasoning, entropy identities, and information-theoretic interpretations. |
|---|
Despite its utility, KL divergence is not symmetric: swapping the roles of \(P\) and \(Q\) generally changes the value. This directionality can be inconvenient when the goal is to measure “overall discrepancy” without preferring one distribution as the reference.
1.2 Symmetrization of directed divergences
Jeffreys’ divergence addresses the asymmetry by combining KL divergences in both directions. The guiding idea is to average (or equivalently sum, up to a constant factor) the discrepancy computed with \(P\) as the reference and the discrepancy computed with \(Q\) as the reference. This yields a symmetric discrepancy measure that responds to differences in either distribution relative to the other.
Jeffreys’ divergence is therefore best understood as a symmetrized form of KL divergence, designed to reflect mutual dissimilarity.
1.3 Information-theoretic interpretation
From an information perspective, KL divergence can be interpreted as the expected number of extra nats (or bits, with base-2 logs) required to encode samples from one distribution using a code optimized for another. Jeffreys’ divergence aggregates this inefficiency in both directions, effectively asking: how much information is “lost” if one uses a model from the other distribution, and vice versa. Because both directions contribute, it captures mismatches that would be missed if only one direction were considered.
2 Definition and Formal Properties
2.1 Mathematical definition for discrete distributions
For discrete probability distributions \(P\) and \(Q\) defined over a shared finite or countable support, Jeffreys’ divergence is defined as \[
| J(P\|Q)=\mathrm{KL}(P\|Q)+\mathrm{KL}(Q\|P). |
|---|
\] If \(P(x)\) and \(Q(x)\) denote the probability mass functions, then \[
| \mathrm{KL}(P\|Q)=\sum_x P(x)\log\frac{P(x)}{Q(x)}. |
|---|
\] Substituting both directions yields a symmetric expression involving log-ratios of the two mass functions.
2.1.1 Computation in terms of probability mass functions
For distributions with common support where both ratios are well-defined, the computation can be written explicitly as \[
| J(P\|Q)=\sum_x P(x)\log\frac{P(x)}{Q(x)}\;+\;\sum_x Q(x)\log\frac{Q(x)}{P(x)}. |
|---|
\]
| Equivalently, the second sum can be expressed by exchanging \(P\) and \(Q\). In practice, terms where \(P(x)=0\) contribute \(0\) to \(\mathrm{KL}(P\|Q)\) by continuity, but terms where \(Q(x)=0\) while \(P(x)>0\) cause the corresponding KL term to diverge to \(+\infty\), which then also drives \(J(P\|Q)\) to \(+\infty\). |
|---|
2.2 Extension to continuous distributions
When \(P\) and \(Q\) are continuous distributions that admit densities \(p\) and \(q\) with respect to a common dominating measure, Jeffreys’ divergence is defined using the same directed KL formula with integrals in place of sums.
2.2.1 Density form and regularity considerations
Under suitable regularity conditions, \[
| \mathrm{KL}(P\|Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx, |
|---|
\] and \[
| J(P\|Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx+\int q(x)\log\frac{q(x)}{p(x)}\,dx. |
|---|
\] The integral form requires that \(p(x)\log(p(x)/q(x))\) and \(q(x)\log(q(x)/p(x))\) be integrable where defined. As in the discrete case, support mismatch (where one density is zero while the other is positive on a set of nonzero measure) can lead to divergence to \(+\infty\).
2.3 Nonnegativity and identity of indiscernibles
KL divergence is nonnegative and equals zero exactly when the two distributions are identical almost everywhere (with respect to the underlying measure). Since Jeffreys’ divergence is the sum of two KL divergences,
| - \(J(P\|Q)\ge 0\), and |
|---|
| - \(J(P\|Q)=0\) if and only if \(P=Q\) almost everywhere. |
Thus Jeffreys’ divergence has the “identity of indiscernibles” property.
2.4 Symmetry and relation to KL divergence
By construction, Jeffreys’ divergence is symmetric: \[
| J(P\|Q)=J(Q\|P). |
|---|
\] It retains the interpretive strength of KL divergence in each direction while removing KL’s asymmetry. In short, it measures mutual distinguishability by summing the directed relative entropies.
3 Connections to Related Divergences
3.1 Jeffreys’ divergence vs. symmetric KL
| The phrase “symmetric KL divergence” is sometimes used to refer directly to Jeffreys’ divergence, since it is literally \(\mathrm{KL}(P\|Q)+\mathrm{KL}(Q\|P)\). However, terminology can vary across fields; some authors reserve that phrase for related symmetrizations (such as averages rather than sums) or for particular normalizations. Conceptually, Jeffreys’ divergence is the canonical symmetric combination of KL in both directions. |
|---|
3.2 Relationship to other f-divergences
Jeffreys’ divergence belongs to a broader family of discrepancy measures generated by convex functions, known as \(f\)-divergences. Many commonly used divergences—such as total variation, Hellinger distance, and \(\chi^2\)-type divergences—can be expressed or bounded using \(f\)-divergence frameworks. This relationship matters because \(f\)-divergences share properties like monotonicity under certain transformations (under appropriate conditions) and can be analyzed using common mathematical tools.
Jeffreys’ divergence is often highlighted for its interpretability as a symmetrized KL quantity rather than solely through its \(f\)-divergence form.
3.3 Comparison with Jensen–Shannon divergence
Jensen–Shannon divergence (JSD) is another symmetric divergence derived from KL divergence, built by comparing each distribution to their mixture. Compared to Jeffreys’ divergence, JSD has two notable characteristics:
- It is always finite for probability distributions on a common space (even when supports differ, depending on formulation).
- It is related to a metric via the Jensen–Shannon distance.
Jeffreys’ divergence, in contrast, can become infinite under support mismatch because at least one KL direction can diverge. This difference makes JSD attractive in settings where distributions may have non-overlapping support.
3.4 Metric-like behavior and limits
Although Jeffreys’ divergence is symmetric and nonnegative, it is not generally a metric: it does not necessarily satisfy the triangle inequality. Still, it behaves “metric-like” in the sense that it grows when distributions separate and equals zero only at equality. In limiting regimes—such as when distributions are close—its behavior aligns with local quadratic approximations (see information-geometry connections below), where it can resemble a squared distance up to scaling factors.
4 Applications in Statistics and Machine Learning
4.1 Model comparison and goodness-of-fit intuition
In statistics, KL divergence is a standard tool for assessing how well one probabilistic model approximates another. Jeffreys’ divergence extends this idea by evaluating discrepancy in both directions. When both distributions can be interpreted as candidate models—e.g., an empirical distribution versus a fitted parametric distribution—Jeffreys’ divergence provides a symmetric notion of mismatch that does not privilege one direction.
This can support intuition in goodness-of-fit analysis: if either model assigns very small probability where the other assigns substantial mass, Jeffreys’ divergence will reflect that in at least one KL term.
4.2 Variational inference and divergence penalties
| Variational inference frequently uses KL divergence to define objectives for approximating a complex posterior distribution with a tractable family. While the standard evidence lower bound often leads to \(\mathrm{KL}(q\|p)\)-type terms, symmetric divergences like Jeffreys’ divergence can appear in alternative formulations that aim to balance mode-seeking and mean-seeking behavior. In practice, using a symmetric divergence may change the character of the approximation, sometimes improving certain aspects of approximation quality at the cost of different optimization properties. |
|---|
4.3 Clustering and distributional similarity
Clustering methods that operate on distributions rather than points—such as mixture models with distributional components or algorithms using distances between learned probability models—can use Jeffreys’ divergence as a similarity or dissimilarity measure. Because it is symmetric, it can be convenient when assigning cluster memberships based on pairwise comparisons between distributional representations.
However, the potential for divergence to be infinite under support mismatch can affect clustering performance, particularly when component distributions do not overlap on the same regions.
4.4 Use in generative modeling and learning objectives
In generative modeling, learning objectives often include divergence measures between a model distribution and a target distribution. While KL divergence and JSD are more common in mainstream architectures, Jeffreys’ divergence can be used in settings where:
- symmetric discrepancy is desired,
- both directions carry meaningful learning signals, and
- one can ensure overlapping support or otherwise regularize to avoid infinite values.
Its role is typically as a penalty term in loss functions or as part of distribution comparison utilities.
5 Computational Aspects
5.1 Estimation from data plug-in approaches
When \(P\) and \(Q\) are unknown and only samples are available, Jeffreys’ divergence can be estimated by estimating each KL term. A basic approach uses plug-in estimators: estimate \(p\) and \(q\) (or the probability masses in discrete settings) from empirical frequencies, substitute into the divergence formula, and compute the result.
Such estimators can be biased, especially in high dimensions or when sample sizes are small relative to support size. The sensitivity to estimation error is a practical consideration in applications.
5.2 Numerical stability and support mismatch
Computational issues often arise from the logarithm of probability ratios. If any estimated probability is zero while the other is nonzero, the corresponding log-ratio may blow up, producing very large values or numerical infinities. Even if true probabilities are nonzero but extremely small, finite-precision arithmetic can cause unstable behavior.
In both discrete and continuous contexts, support mismatch is a conceptual and computational concern: Jeffreys’ divergence inherits the “infinite when KL diverges” behavior from KL divergence.
5.3 Approximations and efficient implementations
Efficient implementation strategies depend on distribution families and available analytic forms. For certain parametric families (e.g., Gaussians in common parameterizations), Jeffreys’ divergence can admit closed-form expressions, reducing computation to evaluating a small number of algebraic terms.
For general distributions, one may rely on:
- Monte Carlo estimates of the KL integrals,
- density-ratio estimation methods,
- variational bounds or approximations that avoid directly computing problematic log-ratios.
The choice depends on whether the distributions are known explicitly, represented via models, or estimated from samples.
6 Theoretical Context in Information Geometry
6.1 Geometric view of divergences
Information geometry studies statistical manifolds and the differential-geometric structures induced by divergence measures. KL divergence is central in this area because it connects to notions of relative entropy and induces geometrical structures such as dual coordinate systems (under appropriate conditions). Jeffreys’ divergence, as a symmetrized combination of KL terms, fits into this framework by capturing mutual distinguishability and often relating to symmetric aspects of the geometry.
While KL divergence is not symmetric, its local behavior and resulting curvature properties influence the geometry that Jeffreys’ divergence also reflects.
6.2 Local behavior via Taylor expansions
A key theoretical perspective is to analyze divergence for distributions that are close. Suppose \(P\) and \(Q\) are indexed by parameters and differ by a small perturbation. Under regularity conditions, KL divergence admits a second-order Taylor expansion whose leading term is typically quadratic in the parameter difference. Since Jeffreys’ divergence adds the two directed KL expansions, it often yields a leading-order quadratic form as well.
This connection explains why, near equality, Jeffreys’ divergence can resemble a squared distance associated with a local metric.
6.3 Links to curvature and distinguishability notions
In information geometry, the curvature and distinguishability between nearby distributions can be related to the Fisher information metric and to higher-order terms in divergence expansions. Symmetric divergences like Jeffreys’ can emphasize how changes are perceived in both directions, potentially offering a more balanced distinguishability criterion than a purely directed KL measure. These theoretical links motivate its use in contexts where local geometry matters, such as manifold-based learning and theoretical analyses of statistical estimators.
7 Special Cases and Examples
7.1 Bernoulli distributions
For Bernoulli distributions \(P=\mathrm{Bern}(p)\) and \(Q=\mathrm{Bern}(q)\), Jeffreys’ divergence becomes a function of \(p\) and \(q\) involving log-odds ratios. Explicitly, \[
| \mathrm{KL}(P\|Q)=p\log\frac{p}{q}+(1-p)\log\frac{1-p}{1-q}, |
|---|
\]
| and \( \mathrm{KL}(Q\|P)\) follows by swapping \(p\) and \(q\). Summing yields \(J(P\|Q)\), which is symmetric in \((p,q)\). |
|---|
This example illustrates typical behavior: divergence grows as \(p\) and \(q\) separate, and it can diverge when one probability is zero and the other is not.
7.2 Gaussian distributions and closed-form expressions
For multivariate normal distributions, Jeffreys’ divergence can be expressed in closed form in terms of the mean vectors and covariance matrices, leveraging known formulas for \(\mathrm{KL}\) between Gaussians. In such settings, the computation reduces to evaluating log-determinants and quadratic forms, enabling fast comparisons between Gaussian models.
This makes Jeffreys’ divergence practical in applications where Gaussian approximations are used, such as uncertainty quantification, probabilistic regression, and Gaussian mixture modeling.
7.3 Degenerate and near-degenerate cases
When distributions become nearly degenerate—such as Gaussians with covariances approaching singularity—numerical issues and conceptual infinities may arise. Although Jeffreys’ divergence can sometimes be computed using limiting arguments, naive implementations may suffer from unstable determinants or large condition numbers. Regularization (e.g., adding small diagonal noise) is commonly used to maintain well-defined covariances and to obtain stable divergence estimates.
Near-degenerate cases also highlight the role of support and overlap: as distributions concentrate, even small mismatches can substantially increase divergence.
8 Practical Guidance and Pitfalls
8.1 When divergence is undefined support issues
Jeffreys’ divergence can be infinite when the support of one distribution is not contained in the support of the other. Since each KL term requires that wherever \(P\) is positive, \(Q\) must also be positive (and similarly for the reverse direction), any region with one distribution assigning probability while the other assigns zero leads to divergence.
For continuous distributions, this manifests as sets of positive measure where one density vanishes but the other does not, potentially making direct use of Jeffreys’ divergence problematic.
8.2 Interpretation of scale and magnitudes
Because Jeffreys’ divergence sums two KL quantities, its magnitude should be interpreted with care. It depends on the chosen log base (affecting units) and on how concentrated the distributions are. Larger values indicate stronger mutual discrepancy, but comparing magnitudes across different problems or variable scalings may be misleading without normalization.
In parametric models, scaling parameters can change divergence values even if qualitative similarity is unchanged, so interpretation often benefits from context-specific calibration.
8.3 Sensitivity to outliers and heavy tails
KL divergence is known to be sensitive to regions where the reference distribution assigns very low probability. Jeffreys’ divergence inherits this sensitivity in both directions, which can amplify the influence of outliers or heavy-tailed behavior. When empirical estimates are contaminated by rare events, divergence computations can become disproportionately large.
This sensitivity motivates robust alternatives or regularization strategies when data are noisy or distributions have heavy tails.
8.4 Choosing between symmetric divergences in practice
In practice, one may choose among Jeffreys’ divergence, Jensen–Shannon divergence, Hellinger-type measures, or other symmetric discrepancies. Jeffreys’ divergence is attractive when:
- distributions have overlapping support,
- direction-agnostic discrepancy is desired, and
- closed forms or stable computation are available.
When support mismatch is likely or finiteness is essential, Jensen–Shannon divergence or other always-finite measures may be preferable. When robustness to outliers is crucial, divergence choices that temper extreme log-ratios can be more stable.
Ultimately, the “best” choice depends on computational stability, expected support overlap, and the intended statistical meaning of discrepancies in the specific application.