1 Definition and Basic Formula
Cross‑entropy measures the expected number of bits required to encode events from one probability distribution using a code that is optimal for another distribution. Formally, for a true distribution \(p\) and an estimated or model distribution \(q\), cross‑entropy is defined as:
\[ H(p,q) = -\sum_{x} p(x) \log q(x) \]
in the discrete case, with an analogous integral form for continuous distributions. The logarithm is typically taken base 2 (yielding bits) or base \(e\) (nats), depending on the context.
1.1 Discrete case
Let \(p\) and \(q\) be probability mass functions over a finite set of events \(\mathcal{X}\). The discrete cross‑entropy is
\[ H(p,q) = -\sum_{x \in \mathcal{X}} p(x) \log q(x). \]
If \(p(x) = 0\) for some \(x\), the term is taken as \(0\) because \(0 \cdot \log q(x) = 0\) for any finite \(q(x)\). The sum is well‑defined provided \(p\) and \(q\) are both valid probability distributions.
1.2 Continuous case
For continuous random variables with probability density functions \(p(x)\) and \(q(x)\) over a domain \(\Omega\), the cross‑entropy becomes
\[ H(p,q) = -\int_{\Omega} p(x) \log q(x) \, dx. \]
This integral is not a true entropy in Shannon’s sense (differential entropy differs by an infinite constant), but the quantity is still useful as a loss function or divergence measure when the domains is fixed.
2 Relationship to Entropy and Kullback–Leibler Divergence
Cross‑entropy is intimately connected to Shannon entropy \(H(p) = -\sum p(x) \log p(x)\) and the Kullback–Leibler divergence \(D_{\text{KL}}(p \parallel q) = \sum p(x) \log \frac{p(x)}{q(x)}\).
2.1 Decomposition formula
A fundamental identity holds:
\[ H(p,q) = H(p) + D_{\text{KL}}(p \parallel q). \]
This follows directly from the definitions:
\[ \begin{aligned} H(p,q) &= -\sum p(x) \log q(x) \\ &= -\sum p(x) \log p(x) + \sum p(x) \log \frac{p(x)}{q(x)} \\ &= H(p) + D_{\text{KL}}(p \parallel q). \end{aligned} \]
Thus cross‑entropy equals the entropy of the true distribution plus the additional bits needed because of the mismatch between \(p\) and \(q\).
2.2 Non‑negativity and equality conditions
Because \(D_{\text{KL}}(p \parallel q) \ge 0\) (Gibbs’ inequality) and \(H(p) \ge 0\) for discrete distributions, cross‑entropy is always at least as large as the entropy of \(p\):
\[ H(p,q) \ge H(p), \]
with equality if and only if \(p(x) = q(x)\) for all \(x\) (almost everywhere in the continuous case). This property makes cross‑entropy a proper scoring rule for probabilistic predictions.
3 Properties of Cross‑Entropy
3.1 Asymmetry
Cross‑entropy is not symmetric in its arguments: \(H(p,q) \neq H(q,p)\) in general. The asymmetry stems from the fact that \(p\) is the reference distribution (the “true” one) while \(q\) is the approximation. Swapping roles yields a different quantity unless both distributions are identical.
3.2 Convexity in q
For a fixed \(p\), the function \(q \mapsto H(p,q)\) is convex over the set of probability distributions. This convexity follows because the negative logarithm is convex and the sum is linear in \(q\). In optimization, this guarantees that finding a minimizer of cross‑entropy (e.g., in maximum likelihood estimation) is a well‑behaved convex problem when the model family is also convex.
3.3 Additivity for independent events
If \(p\) and \(q\) factorize as products of independent distributions over multiple variables, cross‑entropy separates additively. For example, if \(p(x,y) = p_1(x)p_2(y)\) and \(q(x,y) = q_1(x)q_2(y)\), then
\[ H(p,q) = H(p_1,q_1) + H(p_2,q_2). \]
This additivity mirrors that of Shannon entropy and is useful in analyzing composite systems.
4 Estimation of Cross‑Entropy
In practice, the true distribution \(p\) is often unknown, and cross‑entropy must be estimated from data.
4.1 Empirical cross‑entropy
Given a sample of \(N\) independent observations \(\{x_1, x_2, \dots, x_N\}\) drawn from \(p\), the empirical cross‑entropy between the unknown \(p\) and a model \(q\) is
\[ \hat{H}(p,q) = -\frac{1}{N} \sum_{i=1}^N \log q(x_i). \]
This is also the negative log‑likelihood of the data under \(q\), divided by \(N\). It provides an unbiased estimate of \(H(p,q)\) under the assumption that the data are i.i.d.
4.2 Monte Carlo approximation
When the event space is large or continuous, the sum/integral defining cross‑entropy may be approximated by Monte Carlo sampling. If one draws samples \(\{x_i\}\) from \(p\), the estimator
\[ \tilde{H}(p,q) = -\frac{1}{M} \sum_{i=1}^M \log q(x_i) \]
converges to \(H(p,q)\) as \(M \to \infty\) (law of large numbers). This is essentially the same as the empirical cross‑entropy.
4.3 Bias and variance considerations
The empirical cross‑entropy estimator is unbiased, but its variance depends on the spread of \(\log q(x)\). For low‑probability events under \(q\), \(\log q(x)\) can be large in magnitude, inflating variance. In classification settings with many classes, this can lead to unstable estimates, especially for rare classes. Subsampling or smoothing methods are sometimes used to reduce variance.
5 Applications in Statistics and Machine Learning
5.1 Loss function in classification
Cross‑entropy is the most common loss function for probabilistic classifiers. Given true labels (a one‑hot vector from \(p\)) and predicted probabilities (from \(q\)), minimizing cross‑entropy encourages the model to assign high confidence to the correct class.
5.1.1 Binary cross‑entropy
For binary classification, let the true label be \(y \in \{0,1\}\) and the predicted probability of class 1 be \(\hat{y} \in [0,1]\). The binary cross‑entropy (also called log‑loss) is
\[ L(y,\hat{y}) = -y \log \hat{y} - (1-y) \log(1-\hat{y}). \]
It reduces to the negative log‑likelihood of a Bernoulli model.
5.1.2 Categorical cross‑entropy
For multi‑class classification with \(K\) classes, let the true label be a one‑hot vector \(\mathbf{y}\) and the predicted probabilities be \(\hat{\mathbf{y}}\) (summing to 1). The categorical cross‑entropy loss is
\[ L(\mathbf{y},\hat{\mathbf{y}}) = -\sum_{k=1}^K y_k \log \hat{y}_k. \]
This is exactly \(H(\mathbf{y},\hat{\mathbf{y}})\) for the empirical distribution \(\mathbf{y}\). In neural networks, it is typically used with a softmax output layer.
5.2 Model selection and evaluation
Cross‑entropy can be used as a criterion for comparing probabilistic models. Lower cross‑entropy on a held‑out test set indicates better predictive performance. Information criteria such as AIC and BIC are derived from approximations to the expected cross‑entropy (or negative log‑likelihood).
5.3 Connection to maximum likelihood estimation
Maximum likelihood estimation (MLE) seeks parameters \(\theta\) that maximize the likelihood of the data under a model \(q_\theta\). This is equivalent to minimizing the empirical cross‑entropy between the empirical distribution of the data and \(q_\theta\).
5.3.1 Negative log‑likelihood equivalence
Given i.i.d. data \(\{x_i\}\), the likelihood is \(\prod_i q_\theta(x_i)\). The negative log‑likelihood is
\[ -\sum_i \log q_\theta(x_i), \]
which, divided by \(N\), equals the empirical cross‑entropy \(\hat{H}(\hat{p}_{\text{emp}}, q_\theta)\). Thus MLE is exactly the minimization of cross‑entropy relative to the empirical distribution, and the two objectives are interchangeable.
6 Extensions and Variants
6.1 Weighted cross‑entropy
In imbalanced classification, each class can be assigned a weight \(w_k\) to emphasize rarer classes. The weighted cross‑entropy is
\[ L_{\text{weighted}}(\mathbf{y},\hat{\mathbf{y}}) = -\sum_{k=1}^K w_k \, y_k \log \hat{y}_k, \]
where \(w_k\) are positive constants. This modifies the loss to penalise errors on minority classes more heavily.
6.2 Focal loss
Focal loss is a variant designed to down‑weight well‑classified examples and focus on hard misclassified ones. It is defined as
\[ L_{\text{focal}} = -(1-\hat{y}_t)^\gamma \log \hat{y}_t, \]
where \(\hat{y}_t\) is the predicted probability for the true class and \(\gamma \ge 0\) is a focusing parameter. For \(\gamma = 0\), it reduces to binary cross‑entropy. Focal loss has been widely used in object detection to handle extreme class imbalance.
6.3 Generalized cross‑entropy for robust learning
To improve robustness against label noise, generalized cross‑entropy (GCE) modifies the log function. A common formulation uses the Box‑Cox transformation:
\[ L_{\text{GCE}} = \frac{1 - \hat{y}_t^q}{q}, \]
for a parameter \(q \in (0,1]\). As \(q \to 0\), this approaches the standard cross‑entropy; for \(q = 1\), it becomes the mean absolute error (MAE). Tuning \(q\) allows a trade‑off between efficiency and robustness.