1 Definition and Mathematical Formulation

1.1 Likelihood vs. Log-Likelihood

In statistical modeling, a likelihood describes how probable the observed data are under a parameterized model. For observations \(x\) and parameters \(\theta\), the likelihood is written as \(p(x\mid \theta)\). When data are independent, the likelihood of a dataset factorizes into a product over examples, and products of many probabilities can quickly become numerically small.

Taking the logarithm yields the log-likelihood: \[ \log p(x\mid \theta). \] For independent observations, the log-likelihood becomes a sum: \[ \log p(x_{1:n}\mid \theta)=\sum_{i=1}^n \log p(x_i\mid \theta). \] This additive form is not only numerically convenient but also aligns well with gradient-based optimization.

1.2 Negative Log-Likelihood (NLL)

Negative log-likelihood is defined as the negative of the log-likelihood: \[ \text{NLL}(\theta)= -\log p(x\mid \theta). \] For a dataset with independent samples: \[ \text{NLL}(\theta)= -\sum_{i=1}^n \log p(x_i\mid \theta). \] Because training procedures typically minimize a loss, using the negative sign converts maximum-likelihood objectives into minimization problems.

1.3 Relation to Maximum Likelihood Estimation

Maximum likelihood estimation (MLE) chooses parameters that maximize the log-likelihood: \[ \hat{\theta}_{\text{MLE}} = \arg\max_\theta \log p(x\mid \theta). \] Equivalently, because maximizing \(\log p(x\mid \theta)\) is the same as minimizing its negative: \[ \hat{\theta}_{\text{MLE}} = \arg\min_\theta \left(-\log p(x\mid \theta)\right) = \arg\min_\theta \text{NLL}(\theta). \] This equivalence is the core reason NLL is ubiquitous in probabilistic modeling and machine learning.

1.4 Units, Scaling, and Sign Conventions

NLL depends on the logarithm base, and its numerical magnitude therefore carries different units. Also, different software may report summed NLL, average NLL, or NLL scaled by constants, which affects interpretability across experiments.

1.4.1 Natural Log vs. Log Base 10

If the log is natural (\(\ln\)), the NLL is measured in nats (information units using base \(e\)). With log base 10 (\(\log_{10}\)), it is measured in hartleys up to a constant conversion. Since \[ \log_{10}(p)=\frac{\ln(p)}{\ln(10)}, \] changing the base scales NLL by a constant factor of \(1/\ln(10)\). Relative comparisons within the same base remain valid.

2 Common Statistical Models and NLL Forms

2.1 Bernoulli and Binomial NLL

For a Bernoulli model, each observation \(y\in\{0,1\}\) follows: \[ p(y\mid p)=p^y(1-p)^{1-y}, \] where \(p\) is the predicted probability of \(y=1\).

The NLL becomes: \[ \text{NLL}(p)= -\left[y\log p + (1-y)\log(1-p)\right]. \]

2.1.1 Binary Classification Objective

When a classifier outputs \(p=\sigma(z)\) via a logistic sigmoid of a logit \(z\), the NLL is the familiar binary cross-entropy loss. Minimizing it encourages the model to assign high probability to the observed class, with gradients that strongly penalize confident incorrect predictions.

For the binomial distribution (counts with \(n\) trials and \(k\) successes), the likelihood includes a combinatorial factor \(\binom{n}{k}\). In optimization, some implementations drop terms that do not depend on model parameters; the remaining parameter-dependent part still has the same log-probability structure.

2.2 Categorical and Multinomial NLL

In multiclass classification, a categorical distribution models one draw among \(C\) classes with probabilities \(\{p_1,\dots,p_C\}\). For a one-hot target \(y\) with the correct class index \(t\): \[ \text{NLL}= -\log p_t. \] When models produce logits \(z\) and probabilities via softmax \(p_c=\frac{e^{z_c}}{\sum_j e^{z_j}}\), the NLL corresponds to softmax cross-entropy.

For a multinomial distribution with counts \(\{k_1,\dots,k_C\}\) and total \(N\), the likelihood involves terms like \(\prod_c p_c^{k_c}\) (plus a multinomial coefficient independent of parameters). The NLL therefore becomes a weighted sum of \(-\log p_c\) terms: \[ \text{NLL} = -\sum_{c=1}^C k_c \log p_c \; + \; \text{constant}. \]

2.3 Gaussian (Normal) NLL

For continuous targets \(y\) and a Gaussian model: \[ y\sim \mathcal{N}(\mu,\sigma^2), \] the negative log-likelihood is: \[ \text{NLL} = \frac{1}{2}\log(2\pi\sigma^2) + \frac{(y-\mu)^2}{2\sigma^2}. \] This form makes clear how NLL balances two effects: the dispersion term \(\log\sigma^2\) and the squared error term scaled by \(1/\sigma^2\).

2.3.1 Mean-Only vs. Mean-and-Variance Models

If \(\sigma^2\) is fixed, minimizing NLL reduces (up to constants) to minimizing mean squared error (MSE). If \(\sigma^2\) is learned, the model can represent both aleatoric uncertainty (data noise) and heteroscedastic behavior (input-dependent variance), typically improving calibration but also requiring careful constraints to keep \(\sigma^2>0\).

2.4 Poisson NLL

For count data with: \[ y\sim \text{Poisson}(\lambda), \] the NLL is: \[ \text{NLL} = \lambda - y\log\lambda + \log(y!). \] As with other discrete models, the \(\log(y!)\) term does not depend on model parameters when \(\lambda\) is the only parameter, so many training objectives omit it while keeping the parameter-dependent parts.

2.5 Exponential Family Connections

Many common distributions used in machine learning belong to the exponential family, for which log-likelihoods share a structured form. This structural similarity explains why NLL often yields well-behaved gradients and connects naturally to canonical link functions.

2.5.1 Unifying View via Dev iance/Canonical Form

In exponential-family settings, the negative log-likelihood can be expressed using a *deviance* or *canonical* representation. While different distributions have different constants and sufficient statistics, the core idea remains: NLL is tied to how far the model’s natural parameter is from values implied by the observations.

3 Optimization and Training Usage

3.1 Minimizing NLL in Practice

In training, one typically forms NLL over a dataset: \[ \text{NLL}_{\text{total}} = \sum_{i=1}^n -\log p(x_i\mid \theta). \] Optimizers then update \(\theta\) to reduce this quantity. In probabilistic neural networks, parameters might include both network weights and distribution parameters (e.g., \(\mu\) and \(\sigma\) for Gaussian regression).

3.1.1 Gradient Computation Overview

Although the exact derivatives depend on the model, NLL generally yields gradients proportional to the difference between observed outcomes and the model’s assigned probabilities (or moments). For example, in categorical NLL with softmax, the gradient with respect to logits has a simple form: it is the difference between the predicted class probabilities and the one-hot target.

3.2 Regularization and Penalized NLL

Often, raw NLL is insufficient to control overfitting. A penalized objective adds a regularization term: \[ \mathcal{L}(\theta)=\text{NLL}(\theta)+\Omega(\theta), \] where \(\Omega(\theta)\) might be an \(\ell_2\) penalty or other constraint.

3.2.1 L2 (Weight Decay) with Probabilistic Models

For many models, adding \(\frac{\lambda}{2}\|\theta\|_2^2\) corresponds to Gaussian priors on weights under a Bayesian interpretation. While this does not change the likelihood structure of NLL, it modifies the optimum by discouraging overly large parameter values, often improving generalization.

3.3 Numerical Stability Techniques

Naively computing log-likelihoods can fail when probabilities are extremely close to 0 or 1, causing \(\log(0)\) or underflow.

3.3.1 Safe Log-Sum-Exp and Clipping

Common practices include:

  • Using stable implementations of softmax and log-softmax based on the log-sum-exp trick.
  • Avoiding explicit probability multiplication by working in log space.
  • Clipping probabilities or using distribution parameterizations that prevent invalid values (e.g., enforcing positivity of variance via transformations).

3.4 Batch, Mini-Batch, and Averaging Conventions

During stochastic optimization, NLL is computed on batches: \[ \text{NLL}_{\text{batch}} = -\sum_{i\in \mathcal{B}} \log p(x_i\mid \theta). \]

Implementations often divide by batch size \(\mathcal{B}\) to produce an average loss, making learning-rate choices and comparisons across batch sizes more consistent. Reporting can therefore use either sum or mean, which must be accounted for when interpreting magnitudes.

4 Interpretation and Information-Theoretic Meaning

4.1 Surprisal and Unexpectedness

For a realized event \(x\), surprisal is defined as: \[ -\log p(x\mid \theta). \] Thus, NLL for a dataset can be seen as an accumulation of surprisal values. When the model assigns low probability to the observed data, NLL grows, reflecting greater “unexpectedness.”

4.2 Cross-Entropy Relationship

When comparing a true data-generating distribution \(p_{\text{data}}\) with a model distribution \(p_\theta\), the cross-entropy is: \[ H(p_{\text{data}},p_\theta)=\mathbb{E}_{x\sim p_{\text{data}}}\left[-\log p_\theta(x)\right]. \] Empirically, average NLL over samples drawn from \(p_{\text{data}}\) estimates this cross-entropy. In classification tasks, this yields the well-known connection between NLL and cross-entropy loss.

4.3 Expected NLL and Model Calibration

If a model is well-calibrated, its predicted probabilities should match observed frequencies. Because NLL penalizes mismatches in probability mass, it can serve as an objective measure of calibration: overly confident wrong predictions tend to incur large NLL penalties. However, calibration also depends on how well the model class can represent the true conditional distribution.

4.4 Entropy and Divergence Perspectives

The expected negative log-likelihood can be decomposed into components involving entropy and divergence.

4.4.1 Connection to Kullback–Leibler Divergence

A central identity relates cross-entropy to Kullback–Leibler (KL) divergence: \[ \mathrm{KL}(p_{\text{data}}\Vert p_\theta)=\mathbb{E}_{x\sim p_{\text{data}}}\left[\log \frac{p_{\text{data}}(x)}{p_\theta(x)}\right]. \] Rearranging gives: \[ H(p_{\text{data}},p_\theta)=H(p_{\text{data}})+\mathrm{KL}(p_{\text{data}}\Vert p_\theta). \] Since \(H(p_{\text{data}})\) does not depend on \(\theta\), minimizing expected NLL is equivalent to minimizing KL divergence from the data distribution to the model distribution.

5 Model Evaluation and Reporting

5.1 Interpreting NLL Magnitudes

Absolute NLL values are not universal across tasks because they depend on:

  • the chosen distribution (Gaussian vs. categorical, etc.),
  • whether NLL is averaged or summed,
  • the log base and any constant terms retained or omitted,
  • the natural scale of the data and output constraints.

Consequently, NLL is most reliably interpreted through comparisons under identical settings.

5.2 NLL vs. Accuracy and Other Metrics

Accuracy measures correctness of discrete predictions, often ignoring how *confident* the model is when it is right or wrong. NLL captures both correctness and confidence by using the full predicted probability distribution (or density). For probabilistic models, NLL can improve even when accuracy plateaus, reflecting better uncertainty estimates and ranking of outcomes.

5.3 Per-Example NLL and Aggregate Scoring

A common reporting practice is per-example NLL: \[ \text{NLL}_{\text{avg}} = \frac{1}{n}\sum_{i=1}^n -\log p(x_i\mid \theta). \] Per-example scoring makes it easier to compare datasets of different sizes and to track improvements during training.

5.4 Comparing Models with Different Output Sizes

When comparing models that output different numbers of classes or different parameterizations, direct NLL comparison can be misleading. Changes in output dimensionality alter the probability mass structure and may introduce constant offsets. Comparisons should ideally be done under a consistent likelihood definition for the same problem setup.

6.1 Cross-Entropy Loss as NLL

For categorical and Bernoulli settings, cross-entropy loss is effectively NLL. With one-hot targets, cross-entropy reduces to \(-\log p_t\) where \(t\) is the true class, matching the NLL under a corresponding likelihood.

6.2 Mean NLL vs. Sum NLL

Some objectives use sum: \[ -\sum_{i=1}^n \log p(x_i\mid \theta), \] while others use mean: \[ -\frac{1}{n}\sum_{i=1}^n \log p(x_i\mid \theta). \] Both induce the same minimizer if they differ only by a constant factor, but they scale gradients differently during optimization. Reporting should specify which convention is used.

6.3 NLL with Label Smoothing

Label smoothing replaces hard one-hot targets with a softened distribution, for example: \[ y^{\text{smooth}} = (1-\epsilon)\,\text{one-hot} + \epsilon\,u, \] where \(u\) is a uniform or prior distribution and \(0<\epsilon<1\). Using NLL (cross-entropy) with these targets prevents the model from becoming overly confident, often improving generalization and robustness to noisy labels.

6.4 NLL in Variational Inference

In variational inference, optimization typically involves maximizing an evidence lower bound (ELBO) or minimizing a related divergence. The NLL term appears as a reconstruction or likelihood component, while additional terms account for the complexity of the approximate posterior. The resulting objective can be viewed as a structured generalization of NLL-based training.

6.5 Negative Log-Posterior vs. NLL

In Bayesian modeling, one may minimize the negative log-posterior: \[ -\log p(\theta\mid x)= -\log p(x\mid \theta) - \log p(\theta) + \text{constant}. \] Here, the first term is exactly NLL. The extra \(-\log p(\theta)\) term comes from the prior and changes the optimization target compared with pure NLL.

7 Implementation Considerations

7.1 Data Preprocessing Effects on NLL

Preprocessing steps that alter the data distribution can change the appropriate likelihood model. For example, transforming targets for regression (such as scaling or log transforms) affects the implied noise model and thus the correct Gaussian NLL form. If the likelihood is not updated to reflect the transformation, NLL comparisons become inconsistent.

7.2 Handling Missing or Masked Observations

In many applications, certain targets may be missing or intentionally masked. Implementations compute NLL only over observed entries, typically by applying a mask to the per-example contributions: \[ \text{NLL} = -\sum_{i} m_i \log p(x_i\mid \theta), \] where \(m_i\in\{0,1\}\). Averaging is usually adjusted to divide by the number of unmasked observations.

7.3 Dealing with Extreme Probabilities

NLL becomes large when predicted probabilities for observed outcomes approach zero. To mitigate numerical issues and training instability, practitioners may:

  • use log-probability formulations directly,
  • add small epsilons where appropriate,
  • enforce lower bounds on variance parameters for continuous models,
  • apply temperature or calibration techniques in classification settings.

7.4 Framework-Specific Functions and Conventions

Software libraries often provide specialized primitives (e.g., “log-softmax + NLL loss”) that combine steps for stability and efficiency. Differences across frameworks may include:

  • whether they assume logits or probabilities as input,
  • whether NLL terms include constant factors like \(\log(2\pi)\) for Gaussians,
  • default reduction mode (sum vs. mean).

Reliable interpretation requires matching these conventions to the stated likelihood.

8 Worked Examples

8.1 Simple Bernoulli NLL Calculation

Suppose \(y=1\) and the model predicts \(p=0.8\) for the probability of \(y=1\). The Bernoulli NLL is: \[ \text{NLL} = -\log(0.8). \] If instead \(y=0\) with the same \(p=0.8\), then: \[ \text{NLL} = -\log(1-0.8)= -\log(0.2). \] The second case is larger because the model assigns low probability to the actually observed outcome.

8.2 Categorical NLL for Softmax Outputs

Assume 3 classes with softmax probabilities \((p_1,p_2,p_3)=(0.1,0.7,0.2)\), and the true class is class 2. Then: \[ \text{NLL} = -\log(p_2)= -\log(0.7). \] If the model had assigned \(p_2=0.01\) instead, the NLL would increase dramatically, reflecting strong disagreement between predicted and observed classes.

8.3 Gaussian NLL for Regression

Consider a single target \(y=2.0\), predicted mean \(\mu=1.5\), and predicted variance \(\sigma^2=0.25\). The Gaussian NLL is: \[ \text{NLL}=\frac{1}{2}\log(2\pi\cdot 0.25)+\frac{(2.0-1.5)^2}{2\cdot 0.25}. \] Here, the squared error term is \((0.5)^2/(0.5)=0.5\), while the variance term contributes an additional constant depending on \(\sigma^2\). If \(\sigma^2\) increased, the squared error penalty would decrease, but the \(\log\sigma^2\) term would grow.

8.4 Interpreting Changes in NLL During Training

During training, decreasing NLL typically indicates that the model is assigning higher likelihood to the observed data. Improvements may be seen as:

  • NLL decreases on the training set while validation NLL follows,
  • NLL decreases more slowly as the model approaches its representational limits,
  • NLL improves even when accuracy changes little, reflecting more accurate probability estimates.

Conversely, if training NLL decreases but validation NLL increases, this can indicate overfitting or a likelihood mismatch with the true data-generating process.