1 Definition and Motivation
Cross-entropy loss quantifies the mismatch between a predicted probability distribution and a target distribution. In supervised machine learning, the model produces probabilities for outcomes, and the loss assigns a penalty based on how much probability the model places on the correct outcome(s). The penalty increases sharply when the model is confident yet wrong, which makes the measure particularly useful for classification.
1.1 Probability distributions and prediction targets
Let a model output predicted probabilities over classes. For a given training instance, the target is typically represented as either:
- a single correct class (a “hard” target), or
- a distribution over classes (a “soft” target), such as from annotations or teacher models.
Cross-entropy measures the expected negative log probability of the target under the model’s predictions, thereby aligning learning with correct class likelihood.
1.2 Connection to information theory
Cross-entropy is rooted in information theory, where probability distributions are interpreted through the lens of uncertainty and information content.
1.2.1 Entropy and surprise
Entropy captures the average uncertainty of a distribution. A related quantity, often described informally as “surprise,” is the negative log probability of an event: events with low assigned probability correspond to high surprise. Cross-entropy extends this idea by averaging surprise under a target distribution rather than under the model alone.
1.2.2 Kullback–Leibler divergence relationship
Cross-entropy between a target distribution and a model distribution can be decomposed into:
- the target’s entropy, plus
- the Kullback–Leibler divergence from the target to the model.
Because the target entropy term is constant with respect to the model parameters, minimizing cross-entropy is equivalent to minimizing the KL divergence, which directly reduces distribution mismatch.
1.3 Why cross-entropy is suitable for classification
Classification training aims to produce probability estimates that reflect which class is correct, not merely to rank classes correctly.
1.3.1 Penalizing low-probability correct classes
If the correct class receives a probability near zero, the negative log term becomes very large. This creates a strong learning signal that discourages confident incorrect predictions while rewarding probability mass placed on the correct class.
1.3.2 Relation to maximum likelihood estimation
In many common classification setups, minimizing cross-entropy corresponds to maximum likelihood estimation. Under typical model choices (e.g., Bernoulli or categorical likelihoods), the cross-entropy loss is the negative log-likelihood of the observed labels, making it a statistically grounded objective.
2 Mathematical Formulations
Cross-entropy takes different forms depending on whether the task is binary or multi-class, and on whether probabilities or logits are provided.
2.1 Binary cross-entropy
Binary cross-entropy applies to tasks with two outcomes, often encoded as labels 0 and 1.
2.1.1 Bernoulli likelihood view
Assume a Bernoulli model where the prediction is a probability \(p\) of label 1. For target \(y \in \{0,1\}\), the loss is: \[ \text{BCE}(p,y)= -\big[y\log p + (1-y)\log(1-p)\big]. \] This expression matches the negative log-likelihood under the Bernoulli assumption.
2.1.2 Sigmoid outputs and target labels
In neural networks, a sigmoid output is often used to produce \(p\). The BCE loss then directly evaluates the probability assigned to the observed label, combining two logarithmic terms depending on whether \(y\) equals 1 or 0.
2.2 Categorical (multi-class) cross-entropy
For \(K\) classes, the model produces a probability vector over classes, typically from a softmax layer.
2.2.1 One-hot target formulation
With a one-hot target \(y\) where the correct class is \(t\), the loss becomes: \[ \text{CE}(\mathbf{p},y)= -\sum_{k=1}^{K} y_k \log p_k = -\log p_t. \] Thus, only the probability of the correct class contributes to the loss for each instance.
2.2.2 Soft-label formulation
If the target is a distribution \(y_k\) over classes, the loss generalizes to: \[ \text{CE}(\mathbf{p},\mathbf{y}) = -\sum_{k=1}^{K} y_k \log p_k. \] This is common in knowledge distillation and label-uncertainty settings.
2.3 Cross-entropy with logits
Many libraries compute cross-entropy directly from logits (unnormalized scores) to improve numerical stability.
2.3.1 Log-sum-exp trick (numerical stability)
For multi-class logits \(\mathbf{z}\), softmax probabilities are: \[ p_k = \frac{e^{z_k}}{\sum_j e^{z_j}}. \] To compute \(-\log p_t\) stably, implementations use the log-sum-exp identity, which rewrites: \[ \log\sum_j e^{z_j} \] in a way that avoids forming extremely large exponentials.
2.3.2 Avoiding overflow/underflow
Directly exponentiating logits can overflow for large positive values, while subtracting large numbers can underflow. The stabilized formulations keep computations within safe floating-point ranges, reducing the risk of NaNs or infinities during training.
3 Loss Variants and Practical Options
Real-world datasets often require adjustments to the basic loss to reflect class frequencies, annotation uncertainty, or sequence structure.
3.1 Weighted cross-entropy
Weighted cross-entropy introduces scaling factors to the loss terms.
3.1.1 Class imbalance handling
When some classes appear much more frequently than others, an unweighted loss can bias learning toward common classes. Weighting can increase the contribution from rare classes so that the model does not ignore them.
3.1.2 Per-sample weighting
Beyond per-class weighting, one can assign weights per training example, such as when sample importance varies or when resampling strategies require correction.
3.2 Label smoothing
Label smoothing modifies the target distribution so it is not purely one-hot.
3.2.1 Preventing overconfidence
Instead of setting \(y_t=1\) and \(y_{k\neq t}=0\), label smoothing assigns a small probability mass to incorrect classes. This reduces the incentive for the model to drive predicted probabilities to extreme values.
3.2.2 Effect on training targets
The model is trained to match a softened target distribution. As a result, the minimum achievable loss and the calibration behavior of the model can change, often improving robustness and generalization.
3.3 Masked / ignored elements
In tasks like language modeling or sequence labeling, parts of the output may be padding or otherwise invalid.
3.3.1 Sequence modeling and padding masks
Masking ensures that padded tokens do not contribute to the loss. Without masking, the model may learn spurious correlations with padding artifacts.
3.3.2 Selecting valid tokens only
Implementations typically compute token-level losses and then apply a mask to retain only positions that correspond to real data. The reduction step then averages or sums over the valid positions.
3.4 Focal-style extensions (overview)
Focal-style losses modify the weighting of cross-entropy based on how difficult an example is.
3.4.1 Emphasizing hard examples
Examples that the model already predicts with high confidence can be down-weighted. Conversely, those that remain uncertain contribute more, which can be helpful in heavily imbalanced classification.
3.4.2 When to consider alternatives
Such extensions may be considered when standard weighting or resampling does not address persistent class imbalance, or when the learning process is dominated by easy cases.
4 Gradients and Optimization Behavior
Understanding gradients helps explain how cross-entropy drives learning and why it often performs well in classification networks.
4.1 Derivatives with respect to probabilities
For binary cross-entropy, the gradient with respect to the predicted probability reflects how far the model’s probability is from the target label. In general, the negative-log form yields gradients that grow as the model assigns less probability to the correct outcome.
4.2 Gradients for logits
When using softmax (multi-class) or sigmoid (binary) outputs, it is common to compute gradients with respect to logits. In many standard setups, gradients take a relatively simple form involving the difference between predicted probabilities and targets, scaled by the loss’s normalization.
4.3 How cross-entropy shapes learning signals
Cross-entropy produces learning signals that depend strongly on both correctness and confidence.
4.3.1 Learning dynamics and confidence
When predictions are wrong and confident, the loss increases significantly, leading to larger updates that push probability mass toward the correct class. As the model improves, updates typically become smaller because the loss curvature and gradient magnitude reduce.
4.3.2 Common failure modes (e.g., saturation)
If the model architecture or optimization schedule causes logits to become extreme too early, gradient signals can diminish in later training phases, slowing progress. Proper initialization, learning-rate selection, and numerically stable implementations help mitigate these issues.
5 Interpretations and Evaluation
Although trained as a loss function, cross-entropy can also be interpreted as an expected measure of predictive uncertainty.
5.1 Loss values and what they mean
Cross-entropy corresponds to expected negative log-likelihood under a target distribution.
5.1.1 Relating loss to expected log-likelihood
For hard labels, the loss is the average of \(-\log p_t\), where \(p_t\) is the model’s predicted probability for the correct class. Lower values indicate that the model assigns higher probability to correct outcomes across the evaluation set.
5.2 Perplexity as an evaluation metric
In language modeling, perplexity is commonly reported as: \[ \text{perplexity} = \exp(\text{cross-entropy}). \] This transforms the loss from a log-scale to an interpretable multiplicative uncertainty measure, indicating how many choices the model effectively considers plausible per step.
5.3 Calibration vs accuracy
Accuracy measures whether the most likely class matches the label, while cross-entropy evaluates the full probability distribution.
5.3.1 Cross-entropy and probability calibration (high level)
A model can have good accuracy but poor probability calibration if it assigns overconfident probabilities. Cross-entropy penalizes such miscalibration because it reacts not only to the chosen class but to the probability assigned to the correct one.
6 Implementation Details
Practical use depends on framework conventions, tensor shapes, and stable numerical computations.
6.1 PyTorch / TensorFlow typical usage
6.1.1 Built-in functions and expected input formats
Libraries usually provide:
- binary cross-entropy functions that accept either probabilities or logits, and
- categorical cross-entropy functions that accept class indices or one-hot/soft targets.
A key detail is whether the function expects logits (unnormalized) or probabilities, as this determines whether a sigmoid/softmax should be applied externally.
6.2 Numerical stability considerations
6.2.1 Avoiding NaNs and infinities
Cross-entropy involves logarithms, so probabilities at exactly 0 can produce infinities. Stable implementations clamp internally or use log-sum-exp / log-sigmoid formulations. It is still common to monitor training for NaNs, which can arise from invalid activations elsewhere in the network.
6.3 Batch computation and reduction modes
Loss is computed per instance (or per token) and then aggregated.
6.3.1 mean vs sum reductions
Frameworks often support reductions such as mean or sum. Mean reduction makes the effective loss scale less sensitive to batch size, while sum reduction can interact with batch size in optimization and learning-rate tuning.
6.3.2 Impact on learning rate tuning
Because reduction affects the magnitude of gradients, changing reduction mode or batch size can require adjusting learning rates to maintain similar optimization dynamics.
7 Connections to Other Loss Functions
Cross-entropy is part of a broader family of objectives used for supervised learning.
7.1 Mean squared error vs cross-entropy
Mean squared error measures squared differences between predictions and targets. For probabilistic classification, cross-entropy tends to align more naturally with likelihood-based modeling because it directly evaluates log probabilities and preserves the probabilistic interpretation of outputs.
7.2 Hinge loss and margin-based methods (high level)
Hinge losses focus on margins between correct and incorrect classes rather than on probability estimates. While both hinge-style and cross-entropy losses can be used effectively for classification, they differ in whether the model is encouraged to produce calibrated probabilities.
7.3 Relationship to negative log-likelihood
Cross-entropy frequently coincides with negative log-likelihood under standard likelihood assumptions. This connection explains why cross-entropy is often compatible with probabilistic interpretation and maximum likelihood training.
8 Example Workflows (Conceptual)
The following conceptual workflows illustrate typical ways cross-entropy is used during training.
8.1 Training a binary classifier
8.1.1 Preparing targets and thresholds
Targets are encoded as 0/1 labels. The network produces a probability via a sigmoid output or produces logits fed into a stable binary cross-entropy function. During evaluation, a threshold (commonly 0.5, or tuned on a validation set) converts probabilities into predicted labels.
8.2 Training a multi-class classifier
8.2.1 Softmax outputs and one-hot targets
The network outputs logits for each class. Softmax converts them to probabilities implicitly inside the loss function or explicitly before computing the loss. With one-hot targets, cross-entropy reduces to the negative log probability assigned to the correct class.
8.3 Using label smoothing in practice (conceptual)
Label smoothing replaces one-hot targets with a mixture of the correct-class indicator and a uniform (or near-uniform) distribution over classes. Training proceeds with the same cross-entropy computation, but the target distribution is softened, discouraging extreme probability assignments and often improving robustness.