1 Definition and basic concept
Relative entropy is a way to compare two probability distributions by measuring how much one differs from another. It is widely used when a single model is chosen to stand in for a more accurate or unknown distribution. The quantity is not symmetric, so it depends on which distribution is treated as the reference and which is treated as the approximation.
1.1 Informal meaning
Informally, relative entropy describes the “distance” between distributions in terms of information loss. If one distribution is used to encode or predict outcomes generated by another, the relative entropy indicates how inefficient that choice is. A small value suggests that the two distributions are similar, while a larger value indicates a greater mismatch.
1.2 Mathematical definition
Relative entropy is defined using logarithms of ratios of probabilities or probability densities. It compares the expected log difference between the true distribution and an alternative distribution. In most settings, it is written as the Kullback–Leibler divergence.
1.2.1 Discrete distributions
For discrete probability distributions \(P\) and \(Q\) on the same set, the relative entropy of \(P\) with respect to \(Q\) is
\[
| D(P\|Q) = \sum_i P(i)\log\frac{P(i)}{Q(i)}. |
|---|
\]
The sum is taken over all outcomes for which \(P(i)\) is defined, with terms interpreted carefully when some probabilities are zero. The logarithm base determines the unit: base 2 gives bits, while the natural logarithm gives nats.
1.2.2 Continuous distributions
For continuous distributions with densities \(p(x)\) and \(q(x)\), relative entropy is written as
\[
| D(P\|Q) = \int p(x)\log\frac{p(x)}{q(x)}\,dx. |
|---|
\]
This expression depends on the densities with respect to a common underlying measure. It is used in many applications, though extra care is needed because densities can behave differently under changes of variables.
1.3 Relationship to Kullback–Leibler divergence
Relative entropy is commonly identified with Kullback–Leibler divergence. In most modern literature, the two terms refer to the same quantity. The name “relative entropy” emphasizes its connection to information content, while “Kullback–Leibler divergence” highlights its role as a divergence rather than a true geometric distance.
2 Properties
Relative entropy has several basic properties that make it useful in mathematics and applied science. Some of these follow directly from its definition, while others depend on deeper results in information theory and measure theory.
2.1 Non-negativity
Relative entropy is always greater than or equal to zero when it is well defined. It equals zero only when the two distributions match almost everywhere. This means that an exact agreement between the distributions is the unique case of no information loss.
2.2 Asymmetry
Relative entropy is not symmetric. In general,
\[
| D(P\|Q) \neq D(Q\|P). |
|---|
\]
This reflects the fact that using \(Q\) to approximate \(P\) is not equivalent to using \(P\) to approximate \(Q\). Because of this, relative entropy does not satisfy the usual requirements of a metric.
2.3 Conditions for finiteness
The quantity is finite only when the reference distribution assigns positive probability or density wherever the target distribution does. If \(Q\) is too small in places where \(P\) has mass, the logarithm of the ratio may become unbounded. Finite relative entropy often requires overlap between the supports of the two distributions.
2.4 Support mismatch
If \(P\) gives positive probability to an outcome that \(Q\) assigns zero probability, the relative entropy is infinite. This reflects a severe mismatch: the approximating distribution completely misses an event that can actually occur. Such cases are especially important in model selection and statistical inference.
3 Interpretation
Relative entropy admits several interpretations that depend on the field in which it is used. These viewpoints are related, but each emphasizes a different aspect of the same mathematical object.
3.1 Information-theoretic interpretation
In information theory, relative entropy measures the extra coding cost incurred when one distribution is used instead of another. It quantifies inefficiency in compression or transmission. The larger the value, the more information is lost by relying on the wrong probabilistic model.
3.2 Statistical interpretation
In statistics, relative entropy measures how distinguishable two probabilistic models are. It can be viewed as an average discrepancy between the true distribution and a candidate model. For this reason, it appears naturally in likelihood-based estimation, hypothesis testing, and model comparison.
3.3 Thermodynamic interpretation
In thermodynamics and statistical mechanics, relative entropy is linked to entropy production and free-energy differences. It often measures how far a system is from equilibrium with respect to a reference state. This interpretation connects probabilistic uncertainty with physical irreversibility.
4 Variants and related measures
Several closely related quantities are used alongside relative entropy. Some are alternative formulations, while others are distinct divergences that share similar goals.
4.1 Cross-entropy
Cross-entropy measures the expected coding cost of one distribution when another distribution is used to represent it. It is closely related to relative entropy through the identity that cross-entropy equals Shannon entropy plus relative entropy. Because of this, minimizing cross-entropy and minimizing relative entropy often lead to the same optimization problem.
4.2 Shannon entropy
Shannon entropy measures the uncertainty within a single distribution. Relative entropy compares two distributions instead of describing one by itself. The two concepts are connected because relative entropy can be expressed as the difference between cross-entropy and Shannon entropy.
4.3 Jensen–Shannon divergence
The Jensen–Shannon divergence is a symmetric measure built from relative entropy. It averages the divergence from each distribution to a midpoint distribution. Unlike relative entropy, it is bounded and symmetric, which makes it attractive in some practical applications.
4.4 Rényi divergence
Rényi divergence generalizes relative entropy by introducing a parameter that adjusts sensitivity to different parts of the distributions. As the parameter approaches 1, it recovers the Kullback–Leibler form. This broader family is useful when tail behavior or robustness is especially important.
5 Applications
Relative entropy appears in many disciplines because it provides a natural way to compare probabilistic descriptions. It is especially useful whenever uncertainty, approximation, or model selection is involved.
5.1 Data compression
In data compression, relative entropy measures how much extra storage is required when one encoding scheme is based on the wrong probability model. It helps evaluate the efficiency of codes and estimate the penalty for imperfect assumptions. This makes it central to the theory of optimal coding.
5.2 Machine learning
Machine learning methods often optimize objective functions based on relative entropy or its variants. It is used to compare predicted and observed distributions, especially in probabilistic models. The measure appears in supervised learning, generative modeling, and training objectives for neural networks.
5.2.1 Model fitting
During model fitting, relative entropy can quantify the discrepancy between empirical data and a proposed model. Minimizing this discrepancy improves the alignment between predictions and observed frequencies. In many settings, this is equivalent to maximizing likelihood.
5.2.2 Variational inference
Variational inference uses relative entropy to compare an approximate posterior distribution with the exact posterior distribution. The goal is to find a simpler distribution that is still close to the true one. This approach is common when exact Bayesian calculations are too difficult.
5.3 Bayesian statistics
In Bayesian statistics, relative entropy helps assess how much posterior beliefs differ from prior beliefs. It also plays a role in comparing approximate and exact posterior distributions. Because it measures information gain, it is useful for understanding how data update uncertainty.
5.4 Statistical mechanics
In statistical mechanics, relative entropy is used to compare a system’s distribution with an equilibrium or reference distribution. It can express the tendency of systems to move toward states of lower free energy. This makes it valuable in analyses of equilibrium and nonequilibrium behavior.
5.5 Signal processing
In signal processing, relative entropy is used to detect changes in distributions, compare signal models, and identify departures from noise assumptions. It can support classification, anomaly detection, and adaptive filtering. Its sensitivity to mismatches makes it effective in distinguishing competing hypotheses.
6 Mathematical properties
Relative entropy has a rich mathematical structure. Many of its properties arise from convexity and from the way probabilities behave under transformations and conditioning.
6.1 Convexity
Relative entropy is convex in its first argument and jointly convex in both arguments under suitable conditions. This property supports stable optimization and helps explain why minimization problems involving relative entropy often have well-behaved solutions. Convexity is also important in proving inequalities and uniqueness results.
6.2 Chain rule
A chain rule allows relative entropy to be decomposed across conditional distributions. This means that the divergence between joint distributions can be separated into a marginal part and a conditional part. Such decompositions are useful when studying sequences, hierarchies, or multipart systems.
6.3 Data processing inequality
The data processing inequality states that applying a transformation to data cannot increase the relative entropy between two distributions. In other words, any coarse-graining or noisy processing can only reduce distinguishability. This principle is central to information theory and statistical inference.
6.4 Additivity for independent variables
When distributions factor into independent components, relative entropy is additive across those components. The divergence between product distributions equals the sum of the divergences of the factors. This property makes the measure especially convenient for multivariate systems and repeated trials.
7 Generalizations
The basic idea of comparing distributions extends beyond elementary probability theory. More advanced settings preserve the same conceptual goal while adapting the definition to richer mathematical structures.
7.1 Relative entropy in measure theory
In measure-theoretic settings, relative entropy is defined using Radon–Nikodym derivatives with respect to a common measure. This framework allows the concept to apply to general probability spaces, not just finite or smooth ones. It provides the standard foundation for rigorous treatments in analysis and probability.
7.2 Quantum relative entropy
Quantum relative entropy extends the idea to quantum states represented by density matrices. It compares noncommutative probability structures and is used in quantum information theory. The measure plays a role in entanglement theory, quantum hypothesis testing, and open-system dynamics.
7.3 Continuous-time and dynamical formulations
In dynamical systems and stochastic processes, relative entropy can be extended to compare trajectories, path measures, or time-dependent distributions. These formulations are useful for studying evolution, control, and nonequilibrium behavior. They connect static probability comparisons with time-resolved processes.
8 History and development
The concept of relative entropy emerged from several overlapping lines of research in mathematics, physics, and communication theory. Its modern form reflects contributions from information theory and statistical mechanics.
8.1 Early information theory
Relative entropy became prominent through the development of information theory in the mid-20th century. Researchers studied how probability distributions affect coding efficiency and communication reliability. The resulting measure provided a natural way to quantify loss from using an incorrect model.
8.2 Statistical mechanics foundations
Ideas related to relative entropy also appeared in statistical mechanics, where distributions over microscopic states describe macroscopic behavior. The measure helped formalize comparisons between equilibrium states and other probability distributions. This strengthened the link between entropy, uncertainty, and physical systems.
8.3 Modern theoretical uses
In later decades, relative entropy became a standard tool across statistics, machine learning, and applied mathematics. Its role expanded from a specialized theoretical quantity to a practical objective for inference and optimization. Today it is one of the most widely used measures for comparing probabilistic models.