1 Definition and basic idea
In statistics, the likelihood function is a function of model parameters constructed from observed data. For each candidate parameter value, the likelihood reports how plausible the observed data are under the model with those parameters. Although the notation may resemble that of probability densities or mass functions, the interpretation differs: the observed data are treated as fixed, whereas the parameters are treated as unknown and vary.
1.1 Probability versus likelihood
A probability density (or mass) function assigns probabilities to outcomes given fixed parameters. In contrast, likelihood evaluates the model fit by reweighting parameter values according to how well they generate the observed data. This means that two parameter values can be compared by taking the ratio of their likelihoods for the same dataset, even though neither value is itself a probability.
1.2 Likelihood as a function of parameters
Formally, for data \(x\) and parameter vector \(\theta\), a likelihood is written as \(L(\theta\mid x)\). The data \(x\) are fixed, and \(\theta\) is the argument. Different choices of \(\theta\) produce different likelihood values, reflecting the model’s implied compatibility with the data.
1.3 Discrete and continuous data
The likelihood construction parallels the form of the model:
- For discrete data, likelihood uses the probability mass function \(p(x\mid \theta)\).
- For continuous data, likelihood uses the probability density function \(f(x\mid \theta)\).
In both cases, likelihood is typically manipulated via ratios or maximization, so absolute scaling constants usually do not affect relative comparisons.
1.4 Likelihood notation and examples
A standard pattern is that a likelihood equals a density or mass expression with the parameter left as a variable. For example, if \(X\) is modeled as \( \mathcal{N}(\mu,\sigma^2)\), then for observed values \(x_1,\dots,x_n\) the likelihood is the product of normal densities evaluated at the observations, viewed as a function of \(\mu\) and \(\sigma^2\).
2 Mathematical formulation
Likelihood depends on how observations relate to one another and on how the model specifies the data distribution as a function of parameters.
2.1 Likelihood for independent observations
When observations are conditionally independent given parameters, the likelihood factors over observations.
2.1.1 Product form
If \(x_1,\dots,x_n\) are independent given \(\theta\) with density or mass \(f(x_i\mid \theta)\) (or \(p\) in discrete cases), then \[ L(\theta\mid x)=\prod_{i=1}^n f(x_i\mid \theta). \]
2.1.2 Log-likelihood
Maximizing products can be numerically and algebraically awkward, so the log-likelihood \[ \ell(\theta)=\log L(\theta\mid x)=\sum_{i=1}^n \log f(x_i\mid \theta) \] turns the product into a sum. Many theoretical results and algorithms are expressed in terms of \(\ell(\theta)\).
2.1.3 Numerical stability
Products of many small probabilities can underflow in computer arithmetic. Working with log-likelihood values mitigates this issue, and care is taken when evaluating \(\log f(x_i\mid \theta)\) near boundaries where densities can be extremely small.
2.2 Likelihood for dependent data
When observations are dependent even after conditioning on parameters, the likelihood must use the joint distribution.
2.2.1 Joint likelihood
For dependent data \(x_1,\dots,x_n\), the likelihood is \[ L(\theta\mid x)=f(x_1,\dots,x_n\mid \theta), \] where the model supplies the correct joint density or mass function.
2.2.2 Conditional likelihood
In some settings, the joint density can be written using conditional distributions, \[ f(x_1,\dots,x_n\mid \theta)=\prod_{i=1}^n f(x_i\mid x_{1:i-1},\theta), \] so dependence is represented through conditioning structure (common in time-series and state-space models).
2.3 Parameter spaces and constraints
Likelihood is defined on the parameter space implied by the model. Constraints such as \(\sigma>0\), probabilities between 0 and 1, or ordered parameters affect optimization and inference. It is common to handle constraints directly (e.g., optimizing over transformed unconstrained variables) or through constrained optimization routines.
3 Likelihood principles
Likelihood principles connect what can be learned from data to the behavior of the likelihood function.
3.1 Likelihood principle
The likelihood principle states that, for inference about \(\theta\), the relevant information is contained in the likelihood function up to proportionality. In practical terms, if two experiments produce proportional likelihoods, they should lead to equivalent inference under a likelihood-based approach.
3.2 Sufficiency and factorization
Sufficient statistics compress data without losing information about parameters. Under the factorization theorem, a statistic \(T(x)\) is sufficient if the joint density can be written as \[ f(x\mid \theta)=g(T(x),\theta)\,h(x), \] meaning the data contribute to inference only through \(T(x)\) once \(\theta\) is fixed.
3.3 Invariance under reparameterization
Likelihood comparisons do not depend on how parameters are labeled. If \(\phi=g(\theta)\) is a one-to-one transformation, the likelihood as a function of \(\phi\) can be obtained by substitution, and the maximizer maps accordingly. This invariance supports consistent interpretation of parameter estimates under reparameterization.
4 Estimation using likelihood
A central use of likelihood is to estimate parameters by selecting values that maximize agreement between model and observed data.
4.1 Maximum likelihood estimation
The most common likelihood-based estimator is obtained by maximizing the likelihood.
4.1.1 Definition of the MLE
The maximum likelihood estimator (MLE) \(\hat{\theta}_{\text{MLE}}\) is defined as \[ \hat{\theta}_{\text{MLE}}=\arg\max_{\theta} L(\theta\mid x) \] or equivalently \(\arg\max_{\theta}\ell(\theta)\).
4.1.2 Existence and uniqueness
An MLE may not exist if the likelihood has no maximum over the parameter space (for example, when it increases toward the boundary). Even when it exists, it may not be unique, especially in models with ridges or weak identifiability. In such cases, practitioners may use constrained formulations, profile likelihood, or alternative estimators.
4.1.3 Asymptotic properties
Under standard regularity conditions and identifiability, the MLE is consistent and asymptotically normal. Its covariance is often approximated using curvature of the log-likelihood near the optimum, linking estimation accuracy to the second derivatives of \(\ell(\theta)\).
4.2 Profile likelihood
When parameters can be separated into nuisance and interest components, the profile likelihood maximizes over nuisance parameters. If \(\theta=(\psi,\lambda)\) where \(\psi\) is the parameter of interest, the profile likelihood is \[ L_p(\psi)=\max_{\lambda} L(\psi,\lambda). \] This reduces dimensionality and supports confidence construction for \(\psi\).
4.3 Penalized likelihood
Penalized likelihood adds a regularization term to control complexity, especially in high-dimensional or ill-posed problems. The modified objective often takes the form \[ \ell(\theta)-\text{penalty}(\theta). \]
4.3.1 Ridge-type penalties
A ridge penalty typically uses an \(L_2\)-norm of parameters. It shrinks estimates smoothly and is well-suited when many predictors have small effects or when multicollinearity is present. The penalty contributes a quadratic term that improves numerical conditioning.
4.3.2 Lasso-type penalties
A lasso penalty uses an \(L_1\)-norm. It can produce exact zeros in estimated coefficients, enabling variable selection. The non-differentiability at zero often motivates specialized optimization methods.
5 Inference from likelihood
Likelihood does not only estimate parameters; it also supports hypothesis tests and interval estimates.
5.1 Likelihood ratio tests
A likelihood ratio test compares the best fit under a null constraint to the best fit under the full model.
5.1.1 Test statistics
With parameters restricted to \(\theta\in\Theta_0\) under the null and unconstrained \(\theta\in\Theta\) under the alternative, the statistic is often \[ \Lambda = 2\left[\ell(\hat{\theta})-\ell(\hat{\theta}_0)\right], \] where \(\hat{\theta}\) maximizes the likelihood over \(\Theta\) and \(\hat{\theta}_0\) over \(\Theta_0\).
5.1.2 Confidence regions
The same machinery yields likelihood-based confidence regions by inverting the test: a parameter value lies in the region if the likelihood ratio does not exceed a threshold determined by an asymptotic distribution.
5.2 Score tests
A score test uses the gradient of the log-likelihood evaluated at the null estimate. It avoids fitting the full unconstrained model, since only quantities under the null are required. The approach can be computationally attractive when constraints make optimization difficult.
5.3 Wald tests
A Wald test uses the estimate and its standard error from the observed or expected information. It compares the parameter’s deviation from the null using a quadratic form. Wald tests rely on approximations that may be less accurate in small samples or near boundaries.
5.4 Likelihood-based intervals
Intervals can be constructed by likelihood ratio thresholds or by profiling out nuisance parameters. These intervals can adapt to nonlinearity in the parameter-response relationship and often yield better coverage than symmetric approximations in moderately sized samples.
6 Likelihood in common models
Likelihood functions appear in nearly all parametric statistical models. Below are representative forms and interpretations.
6.1 Binomial and Bernoulli models
In the Bernoulli model, each observation is a success or failure with probability \(p\). For \(n\) independent Bernoulli trials with \(s\) successes, the likelihood is proportional to \(p^s(1-p)^{n-s}\). For the binomial model, the likelihood is the corresponding binomial mass function as a function of \(p\).
6.2 Normal models
For independent Gaussian observations \(x_i\sim \mathcal{N}(\mu,\sigma^2)\), the likelihood equals the product of normal densities. Maximization typically yields the sample mean for \(\mu\) and a variance estimate derived from the empirical dispersion (with details depending on whether \(\sigma^2\) is treated as \(\sigma^2\) or includes degrees-of-freedom conventions).
6.3 Poisson models
For count data with \(x_i\sim \text{Poisson}(\lambda)\), the likelihood is proportional to \(\prod_i \lambda^{x_i} e^{-\lambda}\), simplifying to a function of the sum of counts. The MLE for \(\lambda\) is often the sample mean of the observed counts.
6.4 Regression models
In regression, the likelihood is built from a linear predictor passed through a link function, producing a distribution for the response conditional on covariates.
6.4.1 Linear regression
Assuming Gaussian errors with constant variance, the likelihood is based on normal densities around the mean determined by \(X\beta\). Maximizing yields least-squares estimates that coincide with the MLE under the stated assumptions.
6.4.2 Logistic regression
For binary outcomes \(y_i\in\{0,1\}\), logistic regression models \(P(Y_i=1\mid x_i)\) using a logit link. The likelihood is formed from Bernoulli mass functions with success probabilities determined by the logistic function applied to \(x_i^\top\beta\).
6.4.3 Poisson regression
For count responses, Poisson regression links the mean parameter to covariates, typically through an exponential function of a linear predictor. The likelihood is the product of Poisson mass functions with means determined by the regression structure.
7 Advanced topics
Beyond standard likelihood, several extensions broaden applicability or improve robustness.
7.1 Marginal likelihood
In settings with latent variables or hierarchical structure, the marginal likelihood integrates out nuisance or latent components. If a model introduces latent variables \(z\), marginal likelihood sums (or integrates) the joint likelihood over \(z\), producing a likelihood that depends only on parameters of interest.
7.2 Integrated likelihood
Integrated likelihood refers to averaging likelihood-related quantities over uncertainty in nuisance parameters. Depending on context, it may involve integration with respect to a measure that reflects weighting choices, often used to form likelihood-based model comparisons.
7.3 Composite likelihood
Composite likelihood constructs an objective from several lower-dimensional likelihood contributions, such as pairwise or conditional pieces, rather than a full joint likelihood. It can be useful when the full likelihood is intractable or overly complex, trading some efficiency for feasibility.
7.4 Quasi-likelihood
Quasi-likelihood generalizes likelihood by specifying the mean-variance relationship without fully specifying the distribution. Estimation proceeds using estimating equations derived from the quasi-likelihood, providing robustness when the assumed distribution is misspecified but moments are correctly modeled.
7.5 Partial likelihood
Partial likelihood focuses on a subset of information in models where nuisance parameters complicate full likelihood evaluation. A classic example is survival analysis with certain baseline hazards eliminated through conditioning, leaving an objective that depends on regression parameters of interest.
8 Computational methods
Because likelihood-based objectives can be nonconvex or expensive to evaluate, computational strategies are essential.
8.1 Optimization algorithms
Finding the MLE or maximizing a likelihood-related objective typically requires numerical optimization.
8.1.1 Newton-Raphson method
Newton-Raphson updates parameters using the gradient and Hessian (or observed curvature) of the log-likelihood. For twice-differentiable objectives, it can converge quickly near the optimum, though it may require good starting values.
8.1.2 Expectation-maximization algorithm
The expectation-maximization (EM) algorithm alternates between computing expected complete-data log-likelihood (E-step) and maximizing it with respect to parameters (M-step). EM is particularly useful when latent variables are present and direct maximization of the observed-data likelihood is difficult.
8.2 Numerical differentiation
When analytic derivatives are unavailable or unreliable, numerical differentiation approximates gradients and Hessians. Step-size selection is important to balance truncation error and rounding error, especially for log-likelihoods with sharply varying curvature.
8.3 Bayesian computation and likelihood evaluation
Even in Bayesian settings, likelihood evaluation remains central. Many algorithms—such as Markov chain Monte Carlo—repeatedly compute the likelihood (or log-likelihood) for proposed parameter values, so efficient and stable likelihood computations directly affect runtime and convergence.
9 Applications
Likelihood functions underpin a wide range of modeling tasks where data are used to infer parameters or compare models.
9.1 Parameter estimation in science and engineering
In scientific measurement and engineering systems, models often produce observations with uncertainty described by known or parametric distributions. Likelihood-based estimation calibrates model parameters, such as rates, variances, or regression coefficients, from experimental data.
9.2 Model selection
Model selection frequently uses likelihood-based criteria that balance fit against complexity. Approaches may include likelihood ratio tests, information criteria, or likelihood-based cross-validation variants, depending on assumptions and goals.
9.3 Machine learning
Many machine learning algorithms can be interpreted through likelihood. Logistic regression and Poisson regression correspond directly to likelihood maximization, while broader classes of probabilistic models use log-likelihood objectives as loss functions to be optimized.
9.4 Survival analysis
Survival analysis often involves time-to-event data with censoring. Likelihood methods—such as partial likelihood for proportional hazards models—address censoring mechanisms and produce parameter estimates tied to the hazard structure.
10 Related concepts
Likelihood connects naturally to several foundational statistical notions.
10.1 Probability density and mass functions
Likelihood uses the same mathematical objects as densities and mass functions, but the conditioning is treated differently. Densities/masses are interpreted as probabilities for random outcomes given fixed parameters; likelihood treats the data as fixed and compares parameter values.
10.2 Bayesian posterior distribution
In Bayesian statistics, the posterior is proportional to the product of a prior distribution and the likelihood. While likelihood alone supports frequentist estimation, incorporating priors yields a distribution over parameters reflecting both data evidence and prior beliefs.
10.3 Prior and prior predictive distribution
The prior predictive distribution describes the distribution of data implied by combining the model with the prior. Although not equal to the likelihood, it provides a way to assess how plausible observed data would be under prior assumptions.
10.4 Fisher information
Fisher information quantifies the curvature of the log-likelihood on average and governs the precision of estimators. It links to asymptotic variance of MLEs and enters many standard approximations for confidence intervals and hypothesis tests.