Cross-entropy originates from information theory, where it quantifies the inefficiency of encoding events from a true probability distribution using a code optimized for a different, predicted distribution. In this context, the “cost” is measured in bits (or nats, if using natural logarithms). Formally, given a true distribution \(P\) and a predicted distribution \(Q\) over the same event space, the cross-entropy \(H(P,Q)\) is defined as the expected negative log-probability assigned by \(Q\) to events drawn from \(P\).
1.1 Relation to Shannon Entropy and Kullback-Leibler Divergence
Shannon entropy \(H(P)\) measures the minimum average number of bits required to encode events from \(P\) using an optimal code. The Kullback-Leibler (KL) divergence \(D_{\text{KL}}(P \parallel Q)\) measures the extra bits needed when using a code based on \(Q\) instead of \(P\). Cross-entropy connects the two via the identity:
\[ H(P,Q) = H(P) + D_{\text{KL}}(P \parallel Q). \]
Thus, cross-entropy equals the true entropy plus the penalty for mismatched coding. Minimizing cross-entropy with respect to \(Q\) is equivalent to minimizing the KL divergence, as \(H(P)\) is constant when \(P\) is fixed.
1.2 Formal Definition for Discrete and Continuous Distributions
For discrete probability distributions \(P\) and \(Q\) over a finite or countably infinite set \(\mathcal{X}\), cross-entropy is defined as:
\[ H(P,Q) = -\sum_{x \in \mathcal{X}} P(x) \log Q(x). \]
For continuous distributions with probability density functions \(p(x)\) and \(q(x)\) over \(\mathbb{R}^n\), cross-entropy (also called the cross-entropy of two densities) is:
\[ H(p,q) = -\int p(x) \log q(x) \, dx. \]
Both definitions assume the logarithm base determines the unit (2 for bits, \(e\) for nats). The continuous form is not directly analogous to the discrete case due to differential entropy properties but remains useful in modeling and optimization.
Cross-entropy possesses several analytical properties that make it attractive for probabilistic modeling and machine learning.
2.1 Non-negativity and Identity
Cross-entropy is always non-negative: \(H(P,Q) \ge 0\). Equality holds if and only if \(P = Q\) and \(H(P)=0\), which occurs only when \(P\) is a Dirac delta distribution (a single event with probability 1). Otherwise, cross-entropy is strictly positive. This non-negativity follows from Gibbs’ inequality or from the non-negativity of KL divergence combined with the non-negativity of Shannon entropy.
2.2 Convexity and Optimization
For a fixed \(P\), the cross-entropy \(H(P,Q)\) is a convex function of the parameters that define \(Q\), provided the log-probabilities are linear in those parameters (e.g., in logistic regression or neural networks with softmax). This convexity ensures that gradient‑based optimization converges to a global minimum when the model is linear in parameters. In deep learning, the loss may not be globally convex, but cross-entropy’s local convexity often aids convergence.
2.3 Asymmetric Behavior
Cross-entropy is not symmetric: in general, \(H(P,Q) \neq H(Q,P)\). This asymmetry mirrors that of KL divergence. In machine learning, the true distribution \(P\) is typically the data distribution (or a one‑hot label vector), and \(Q\) is the model’s prediction. The asymmetry means that cross‑entropy heavily penalizes confident but wrong predictions (when \(Q\) assigns low probability to a high‑probability event under \(P\)).
Cross-entropy finds widespread use in fields ranging from machine learning to communication theory.
3.1 Machine Learning
In supervised learning, cross-entropy serves as a standard loss function for classification models, measuring the discrepancy between predicted class probabilities and true labels.
3.1.1 Cross-Entropy Loss for Classification
The cross-entropy loss is derived by treating the true label distribution as a one‑hot vector: \(P\) assigns probability 1 to the correct class and 0 elsewhere. Minimizing \(H(P,Q)\) then forces the model to assign high probability to the true class.
3.1.1.1 Binary Cross-Entropy
For binary classification (two classes), the loss for a single example is:
\[ L = -[y \log \hat{y} + (1-y) \log (1-\hat{y})], \]
where \(y \in \{0,1\}\) is the true label and \(\hat{y} \in (0,1)\) is the model’s estimated probability of class 1. This is equivalent to the negative log‑likelihood of a Bernoulli random variable.
3.1.1.2 Categorical Cross-Entropy
For multi‑class classification (\(K\) classes), the loss is:
\[ L = -\sum_{i=1}^{K} y_i \log \hat{y}_i, \]
where \(y_i\) is 1 for the true class and 0 otherwise (one‑hot encoding), and \(\hat{y}_i\) is the predicted probability for class \(i\). This generalizes to cases where \(y_i\) can be soft labels (e.g., label smoothing).
3.1.2 Connection to Maximum Likelihood Estimation
Minimizing cross‑entropy loss is mathematically equivalent to maximizing the log‑likelihood of the observed labels under the model. For a dataset of \(N\) independent samples, the average negative log‑likelihood is exactly the empirical cross‑entropy between the empirical data distribution and the model. This connection justifies cross‑entropy as a principled objective for probabilistic models.
3.1.3 Use in Neural Networks with Softmax Output
In neural network classifiers, the final layer typically uses a softmax activation to produce a valid probability distribution over classes:
\[ \hat{y}_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}. \]
The combination of softmax and categorical cross‑entropy yields a convex loss with respect to the linear inputs \(z_i\) (logits). Its gradient has a simple form: \(\partial L / \partial z_i = \hat{y}_i - y_i\), which facilitates efficient backpropagation.
3.2 Communication Theory and Coding
Cross‑entropy underpins several key ideas in information‑theoretic communication and data compression.
3.2.1 Channel Coding and Error Correction
In channel coding, cross‑entropy is used to evaluate the mismatch between the actual channel input distribution and the distribution assumed by the encoder. For a given channel, the capacity is the maximum mutual information; cross‑entropy provides a bound on the achievable rate when the input distribution is suboptimal. It also appears in the analysis of decoding algorithms (e.g., in iterative belief propagation).
3.2.2 Data Compression and Model Evaluation
In lossless compression, the cross‑entropy between the true data distribution and a probabilistic model gives the average number of bits per symbol required to compress data under that model. This is used as a measure of model quality: a lower cross‑entropy indicates a better fit and more efficient compression. Practical algorithms (e.g., arithmetic coding) implement codes that achieve near‑cross‑entropy compression rates.
Several modifications of the standard cross‑entropy loss have been developed to address imbalanced data, hard examples, or overfitting.
4.1 Weighted Cross-Entropy
In classification tasks with class imbalance, each class can be assigned a weight to offset the prevalence of majority classes. The weighted cross‑entropy loss is:
\[ L = -\sum_{i=1}^{K} w_i \, y_i \log \hat{y}_i, \]
where \(w_i\) are class‑specific weights (often inversely proportional to class frequency). This encourages the model to pay more attention to rare classes.
4.2 Focal Loss
Focal loss (proposed in the context of object detection) addresses hard‑to‑classify examples by down‑weighting the loss for well‑classified examples. It modifies the cross‑entropy term with a focusing parameter \(\gamma \ge 0\):
\[ L = -\sum_{i=1}^{K} y_i (1 - \hat{y}_i)^\gamma \log \hat{y}_i. \]
When \(\gamma=0\), focal loss reduces to cross‑entropy. Higher values of \(\gamma\) emphasize training on hard, misclassified examples.
4.3 Label Smoothing and Regularization
Label smoothing replaces the hard one‑hot labels with a mixture of the true label and a uniform distribution:
\[ y_i' = (1 - \varepsilon) y_i + \frac{\varepsilon}{K}, \]
where \(\varepsilon\) is a small smoothing parameter (e.g., 0.1). Cross‑entropy is then computed using \(y_i'\) instead of \(y_i\). This reduces overfitting and improves calibration of neural networks by preventing the model from assigning infinite logits to training examples.