1 Definition and notation

Kullback–Leibler (KL) divergence, typically written as \(D_{\mathrm{KL}}(P\|Q)\), is a quantity that compares two probability distributions \(P\) and \(Q\) defined on the same measurable space. It measures the expected logarithmic discrepancy between the probability mass (or density) that \(P\) assigns and the corresponding amount assigned by \(Q\), where the expectation is taken under \(P\).
Formally, in settings where the divergence is defined, it is expressed as an expectation of a log-likelihood ratio. The asymmetry of the ordering—\(D_{\mathrm{KL}}(P\|Q)\) versus \(D_{\mathrm{KL}}(Q\|P)\)—is central: exchanging the arguments generally changes the value.

1.1 Discrete case

1.1.1 Summation form

For two discrete probability distributions \(P\) and \(Q\) over a common countable set \(\mathcal{X}\), \[

D_{\mathrm{KL}}(P\|Q)=\sum_{x\in\mathcal{X}} P(x)\log\frac{P(x)}{Q(x)},

\] where the logarithm base determines the unit (e.g., base 2 gives bits, base \(e\) gives nats).

1.1.2 Interpretation as expected log-ratio

The expression can be viewed as an expectation under \(P\): \[

D_{\mathrm{KL}}(P\|Q)=\mathbb{E}_{X\sim P}\left[\log\frac{P(X)}{Q(X)}\right].

\] Thus, KL divergence is the average, over outcomes likely under \(P\), of how much \(Q\) under- or overestimates the probability relative to \(P\).

1.2 Continuous case

1.2.1 Integral form

For continuous distributions with densities \(p(x)\) and \(q(x)\) with respect to a shared base measure (commonly Lebesgue measure), KL divergence takes the form \[

D_{\mathrm{KL}}(P\|Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx.

\] This integrates the same local log-ratio, weighted by how frequently \(P\) assigns probability to each region.

1.2.2 Density and support considerations

In continuous settings, the meaning of \(p(x)\) and \(q(x)\) depends on the chosen reference measure and, more importantly, on whether the supports align. If \(Q\) assigns very small density where \(P\) has non-negligible mass, the log-ratio can become large, which is reflected in a larger divergence.

1.3 Conditions for finiteness

KL divergence may be infinite. Determining when it is finite is largely a matter of how \(P\) behaves on regions where \(Q\) assigns zero probability (or zero density).

1.3.1 Absolute continuity

A common sufficient condition is absolute continuity of \(P\) with respect to \(Q\). Informally, whenever \(P\) assigns probability to a set, \(Q\) must assign positive probability (or positive density in a suitable sense) to that same set. When this fails, the integrand includes terms that effectively compare a positive value to zero.

1.3.2 Handling zeros and infinities

In the discrete case, the convention \(0\log 0 = 0\) is typically used. However, if \(P(x)>0\) and \(Q(x)=0\), then the term \(\log(P(x)/Q(x))\) is infinite and so the overall divergence becomes \(+\infty\). In continuous form, an analogous phenomenon occurs when \(q(x)=0\) on regions where \(p(x)>0\) with nonzero measure.

2 Properties of KL divergence

KL divergence has several foundational properties that explain both its widespread theoretical role and its practical behavior.

2.1 Non-negativity and equality conditions

2.1.1 Gibbs’ inequality

A central result is that KL divergence is always non-negative: \[

D_{\mathrm{KL}}(P\|Q)\ge 0.

\] This follows from Gibbs’ inequality, which is derived using convexity arguments for the logarithm. Intuitively, the log-ratio term averages to something that cannot be negative when computed under \(P\).

2.1.2 When KL equals zero

KL divergence equals zero only when \(P\) and \(Q\) agree almost everywhere (with respect to \(P\), and in discrete settings, on all points where \(P\) has positive probability). If there is any set with positive \(P\)-probability where the corresponding values under \(Q\) differ, the divergence becomes strictly positive.

2.2 Non-symmetry

2.2.1 Relationship to reverse KL

In general, \[

D_{\mathrm{KL}}(P\|Q)\neq D_{\mathrm{KL}}(Q\|P).

\]

The “forward” divergence \(D_{\mathrm{KL}}(P\|Q)\) weights the log-ratio by \(P\), while the “reverse” divergence weights it by \(Q\). As a result, the two quantities emphasize different regions: forward KL focuses on where \(P\) is large, whereas reverse KL pays more attention to areas where \(Q\) places mass.

2.3 Not a metric

2.3.1 Lack of triangle inequality

Although KL divergence resembles a distance, it does not satisfy the metric axioms. In particular, the triangle inequality generally fails, and the value is not symmetric.

2.3.2 Practical implications

Because KL is not a metric, it behaves differently from Euclidean or Wasserstein distances in geometry-based analyses. Algorithms that “minimize KL” typically correspond to particular optimization objectives rather than searching for a nearest distribution in a distance sense.

3 Connections to information theory

KL divergence is closely related to fundamental quantities such as entropy, cross-entropy, and mutual information. These connections give it an operational interpretation.

3.1 Relation to entropy

3.1.1 Entropy as a special case

Let \(H(P)\) denote entropy of \(P\). KL divergence can be rearranged in a way that reveals entropy as part of the expression: \[

D_{\mathrm{KL}}(P\|Q)=H(P,Q)-H(P),

\] where \(H(P,Q)\) is cross-entropy. When \(Q=P\), the divergence becomes zero because cross-entropy matches entropy.

3.1.2 Cross-entropy connection

Cross-entropy \(H(P,Q)\) measures the expected negative log-likelihood of \(Q\) under samples from \(P\). Minimizing KL divergence with respect to \(Q\) is therefore equivalent to minimizing cross-entropy, since \(H(P)\) does not depend on \(Q\).

3.2 Mutual information

3.2.1 KL form of mutual information

Mutual information between random variables can be expressed using KL divergence. One common formulation compares a joint distribution to the product of its marginals: \[

I(X;Y)=D_{\mathrm{KL}}(P_{X,Y}\|P_X P_Y).

\] This interpretation views mutual information as the amount by which the joint distribution departs from independence.

3.3 Coding and expected redundancy

3.3.1 Interpretation via optimal coding

In coding theory, KL divergence quantifies the extra expected code length that arises when code lengths are designed for \(Q\) while the true source distribution is \(P\). More precisely, if an optimal code corresponds to probabilities from \(Q\), then the expected inefficiency relative to the optimal code for \(P\) is tied to \(D_{\mathrm{KL}}(P\|Q)\).

4 Estimation and statistical use

KL divergence is routinely used for model comparison, parameter fitting, and learning objectives because it connects directly to likelihood and estimation criteria.

4.1 Maximum likelihood and KL minimization

4.1.1 Projection viewpoint

Suppose \(Q_\theta\) is a parametric family and the goal is to choose \(\theta\) so that \(Q_\theta\) resembles a target distribution \(P\) in KL divergence. Minimizing \(D_{\mathrm{KL}}(P\|Q_\theta)\) is often interpreted as an information projection of \(P\) onto the model family. In discrete exponential families, this projection can correspond to moment matching under certain conditions.

4.2 Empirical estimation from samples

4.2.1 Plug-in estimators

When \(P\) is unknown but samples from \(P\) are available, practitioners may estimate \(D_{\mathrm{KL}}(P\|Q)\) by replacing \(P\) with its empirical distribution (the “plug-in” approach). For discrete variables, this yields a sum over observed frequencies, substituting estimates \(\hat{P}(x)\) into the formula.

4.2.2 Bias and variance considerations

Plug-in KL estimators can be biased, especially in high-dimensional or small-sample regimes, because the empirical distribution can assign zero probability to outcomes that are rare under the sample but nonzero under the true \(P\). The resulting variance can also be substantial, and the bias behavior depends on support size and smoothing choices.

4.3 Model selection perspective

In likelihood-based model selection, differences in log-likelihoods can be related to KL divergence. While model comparison is often expressed through criteria such as deviance or information criteria, the underlying rationale is frequently that models that minimize an expected divergence from the truth tend to generalize better. Asymptotic analyses connect these criteria to how well candidate models approximate the data-generating process.

5 Computational and numerical aspects

Computing KL divergence depends on both the distribution forms and numerical care. Many common cases admit closed forms, while others require approximations.

5.1 Closed forms for common distributions

Closed-form expressions are valuable because they improve speed and accuracy and reduce sensitivity to numerical errors.

5.1.1 Bernoulli and categorical

For Bernoulli distributions with parameters \(p\) and \(q\), \[

D_{\mathrm{KL}}(\text{Bern}(p)\|\text{Bern}(q))=p\log\frac{p}{q}+(1-p)\log\frac{1-p}{1-q},

\] with standard conventions when probabilities are 0 or 1. For categorical distributions, KL divergence generalizes to a finite sum over classes, each term using the log-ratio of class probabilities.

5.1.2 Gaussian (univariate and multivariate)

For univariate Gaussians \(N(\mu_0,\sigma_0^2)\) and \(N(\mu_1,\sigma_1^2)\), KL divergence has an analytic expression involving the mean difference and the ratio of variances. In the multivariate case, the formula uses covariance matrices, combining a trace term, a Mahalanobis-like mean term, and a determinant ratio.

5.1.3 Exponential family examples

For distributions in the exponential family, KL divergence often simplifies to expressions involving sufficient statistics and log-partition functions. These closed forms are particularly useful because they align with the structure used in maximum likelihood and variational methods.

5.2 Approximations and regularization

When closed forms are unavailable or unstable, approximations and constraints are used to make KL computation feasible.

5.2.1 Smoothing to avoid zero-probability issues

A frequent computational challenge is zero probabilities in \(Q\) where \(P\) assigns mass. Smoothing modifies \(Q\) (or the empirical estimate of \(P\)) by adding small pseudo-counts or using regularized density estimates, preventing infinite divergence and improving numerical stability.

5.2.2 Monte Carlo estimation

If \(P\) permits sampling but integrals are difficult, KL divergence can be estimated using Monte Carlo: \[

D_{\mathrm{KL}}(P\|Q)=\mathbb{E}_{X\sim P}\left[\log p(X)-\log q(X)\right].

\] Samples drawn from \(P\) yield empirical averages of the log-density ratio.

5.3 Numerical stability

5.3.1 Log-sum-exp techniques

KL computations often involve expressions that combine logs of probabilities or exponentials. The log-sum-exp trick helps avoid underflow or overflow by computing sums in a stable logarithmic manner, especially when probabilities are extremely small or when model outputs are produced as unnormalized scores.

KL divergence is part of a broader family of divergence measures. Variants differ in directionality, symmetry, and mathematical properties.

6.1 Reverse KL divergence

6.1.1 Distinct behavior in optimization

Reverse KL divergence \(D_{\mathrm{KL}}(Q\|P)\) emphasizes regions where \(Q\) assigns probability. This often leads to different optimization dynamics compared with forward KL. In approximate inference, reverse KL can encourage mode-seeking behavior, while forward KL can encourage broader coverage depending on the context and constraints.

6.2 Symmetrized KL

6.2.1 Jeffreys’ divergence

A simple symmetrization is Jeffreys’ divergence: \[

J(P,Q)=D_{\mathrm{KL}}(P\|Q)+D_{\mathrm{KL}}(Q\|P).

\] It retains sensitivity to discrepancies in both directions but remains a divergence rather than a metric because symmetry does not ensure triangle inequality.

6.3 Jensen–Shannon divergence

6.3.1 Metric properties and boundedness

Jensen–Shannon divergence (JS divergence) symmetrizes KL by comparing each distribution to their average. It is bounded and, unlike KL, has stronger metric-like properties under square-root transformation. JS divergence is often preferred in applications that require symmetry and finiteness even when one distribution assigns zeros where the other does not.

6.4 Other f-divergences

6.4.1 Total variation and χ² divergence

Many divergences can be expressed using a convex function applied to a likelihood ratio, forming the family of f-divergences. Total variation captures the maximal difference in probabilities assigned to events, while χ² divergence measures a squared relative deviation that can be more sensitive to mismatches in regions where the reference distribution is small.

6.4.2 Wasserstein distance (contrast)

Wasserstein distance compares distributions via the cost of transporting mass and reflects geometry in the underlying space. Unlike KL, it remains finite under broader conditions and often produces qualitatively different behavior, especially when distributions are displaced or have partially overlapping supports.

7 Applications in machine learning and inference

In machine learning, KL divergence appears in learning objectives, probabilistic modeling, and approximate inference procedures.

7.1 Variational inference

7.1.1 ELBO and KL terms

Variational inference introduces a tractable approximation \(q(z)\) to a complicated posterior \(p(z\mid x)\). The evidence lower bound (ELBO) contains a KL divergence term of the form \(D_{\mathrm{KL}}(q(z)\|p(z\mid x))\). Maximizing the ELBO is equivalent to minimizing this KL divergence, thereby improving the approximation.

7.2 Expectation–maximization viewpoint

7.2.1 E-step and divergence minimization (intuition)

In expectation–maximization (EM), one alternates between inferring latent variables and updating parameters. The E-step can be interpreted using KL divergence between an auxiliary distribution and the current posterior, while the M-step updates parameters to better fit the data under the inferred latent structure.

7.3 Regularization in learning

7.3.1 KL penalties and constraints

KL divergence is used as a regularizer in many learning algorithms, for example by penalizing deviation from a prior distribution or from a reference model. In Bayesian contexts, KL penalties can play the role of encouraging simpler or more plausible distributions, often improving generalization.

7.4 Distribution alignment and representation learning

7.4.1 Latent-variable modeling intuition

When models include latent variables, KL terms often align an approximate latent distribution with a chosen prior, thereby preventing the latent space from becoming arbitrary. This alignment can yield representations that are easier to sample from and can stabilize training in generative modeling.

8 Theoretical results

KL divergence admits useful theoretical properties in optimization, statistics, and measure theory.

8.1 Convexity properties

8.1.1 Convex in one argument

KL divergence is convex in its first argument \(P\) under standard conditions, while it is generally not convex in the second argument \(Q\). This asymmetry influences how objective functions behave when optimized with respect to parameters governing \(P\) or \(Q\).

8.2 Relation to likelihood ratio tests

8.2.1 Asymptotic interpretations

Log-likelihood ratio statistics are closely tied to KL divergence. Under suitable regularity conditions, divergences between the true and hypothesized distributions govern the asymptotic distribution of such test statistics. This connection motivates using KL-based reasoning in large-sample regimes.

8.3.1 Radon–Nikodym derivative

The most general definition uses the Radon–Nikodym derivative. If \(P\) is absolutely continuous with respect to \(Q\), then the divergence can be written as \[

D_{\mathrm{KL}}(P\|Q)=\int \log\left(\frac{dP}{dQ}\right)\,dP.

\] This formulation clarifies that KL divergence depends on how probability measures relate, not merely on pointwise densities in a specific coordinate system.

9 Practical guidance and common pitfalls

Proper use of KL divergence requires attention to supports, direction, discretization, and interpretation.

9.1 Support mismatch problems

9.1.1 When KL becomes infinite

If the reference distribution \(Q\) assigns zero probability to an event that has positive probability under \(P\), then KL divergence becomes infinite. In applications, this can occur due to limited training data, sparse categorical probabilities, or vanishing density in continuous models, and it can break optimization or distort comparisons.

9.2 Direction choice (forward vs reverse)

9.2.1 Typical outcomes in optimization

Choosing between \(D_{\mathrm{KL}}(P\|Q)\) and \(D_{\mathrm{KL}}(Q\|P)\) changes what the optimization prioritizes. Forward KL tends to penalize missing mass where \(P\) is high, often encouraging the approximation to cover the main support of \(P\). Reverse KL often reacts strongly to regions where the approximating distribution \(Q\) is confident, which can lead to focusing on dominant modes.

9.3 Discretization and binning effects

9.3.1 Continuous-to-discrete approximations

When approximating continuous distributions by discretizing into bins, the estimated KL divergence depends on bin widths and bin alignment. Refining the discretization can change the numerical value and may introduce instability if some bins receive few samples. Careful calibration of binning or use of continuous estimators is often necessary for reliable conclusions.

9.4 Interpretation cautions

9.4.1 KL as “relative information,” not similarity score

KL divergence is not a bounded similarity metric. A small KL indicates that, on average under \(P\), using \(Q\) incurs little informational loss, but it does not guarantee pointwise closeness of distributions. Conversely, large KL can reflect mismatches in regions that are irrelevant for some tasks, so the divergence should be interpreted in light of the modeling objective and the chosen direction.