1 Definition and Intuition
1.1 Likelihood as a Measure of Plausibility
In statistics, likelihood is a function that scores how compatible different values of unknown parameters are with observed data. Instead of asking how likely the data are to occur under a fixed parameter value, the likelihood view treats the data as given and lets the parameters vary. The resulting likelihood values indicate which parameter settings better explain the observations, without claiming that any particular likelihood value is itself a probability of the parameter.
A common way to interpret this idea informally is: if you were to choose a parameter value that best “matches” the data according to a chosen statistical model, the likelihood function provides the quantitative criterion.
1.2 Likelihood Versus Probability
Likelihood and probability are related but used differently. Probability is typically interpreted as a rule for random outcomes when variables are allowed to vary. Likelihood is constructed from the same mathematical expressions but is interpreted as a relative score over parameter values, given the data.
This distinction becomes especially important in continuous models, where likelihood values are density-like quantities. A likelihood function can be compared across parameter values (often via ratios or maxima), but its absolute magnitude generally does not have the direct probabilistic meaning that a probability distribution does for a random variable.
1.3 Notation and Core Form (Likelihood Function)
Consider a statistical model with observed data \(x\) and parameter \(\theta\). A likelihood function is typically written as \[ L(\theta \mid x) \propto f(x \mid \theta), \] where \(f(x \mid \theta)\) is a probability mass function (discrete case) or probability density function (continuous case). The proportionality sign emphasizes that multiplying by factors that do not depend on \(\theta\) does not change comparisons over \(\theta\).
Thus, \(x\) is treated as fixed, and the dependence is on \(\theta\). For likelihood-based methods, such as maximum likelihood estimation, the value of \(L(\theta \mid x)\) across different \(\theta\) is what matters.
2 Mathematical Foundations
2.1 Likelihood from a Parametric Model
A parametric model specifies a family of distributions indexed by parameters. Once the data are observed, the likelihood function reuses that family while reversing the roles of “random” and “unknown.”
2.1.1 Discrete Data Likelihood
For discrete observations \(x\), with probability mass function \(p(x \mid \theta)\), the likelihood is commonly defined as \[ L(\theta \mid x) = p(x \mid \theta), \] or equivalently any expression proportional to it. If observations are independent given \(\theta\), the likelihood for a dataset \((x_1,\dots,x_n)\) becomes the product \[ L(\theta \mid x_1,\dots,x_n)=\prod_{i=1}^n p(x_i \mid \theta). \] This multiplicative structure is a standard source of computational and numerical issues, motivating log-likelihood in later sections.
2.1.2 Continuous Data Likelihood
For continuous observations, with density \(f(x \mid \theta)\), the likelihood function is \[ L(\theta \mid x) = f(x \mid \theta), \] up to any multiplicative constant not involving \(\theta\). For independent continuous observations, the likelihood is again a product of densities: \[ L(\theta \mid x_1,\dots,x_n)=\prod_{i=1}^n f(x_i \mid \theta). \] Because densities are not probabilities of single points, likelihood values should be interpreted comparatively rather than as absolute probabilities.
2.2 The Likelihood Function and Its Domain
The likelihood function is defined on the parameter space. Its domain is the set of parameter values that correspond to valid models. For example, variance parameters must typically be positive, probabilities must lie in \([0,1]\), and shape parameters may need to satisfy additional constraints depending on the distribution.
The domain matters because optimization routines and theoretical properties such as existence of maxima depend on what values of \(\theta\) are allowed.
2.3 Log-Likelihood and Numerical Stability
2.3.1 Why Log-Likelihood Is Common
Products of many probabilities or densities can underflow in computer arithmetic, producing zeros even when the true value is nonzero. Taking logs transforms products into sums: \[ \ell(\theta\mid x)=\log L(\theta\mid x). \] Since the logarithm is monotone, maximizing \(L(\theta\mid x)\) over \(\theta\) is equivalent to maximizing \(\ell(\theta\mid x)\). Log-likelihood therefore improves numerical stability and often simplifies derivative-based optimization because it turns multiplicative terms into additive ones.
3 Properties of Likelihood Functions
3.1 Scaling and Proportionality
Likelihood functions are often treated up to proportionality. If \(L_1(\theta)=c\cdot L_2(\theta)\) where \(c\) does not depend on \(\theta\), both yield the same maximizing parameter value and the same likelihood ratios. This means that constants depending only on the observed data can be ignored in most estimation tasks.
However, proportionality does not justify interpreting likelihood values as probabilities. The key operational use is comparison across parameter values, not normalization over \(\theta\).
3.2 Maximization and Identifiability
Many inference procedures center on maximizing the likelihood. If the likelihood has a unique maximum at \(\hat{\theta}\), that estimate is straightforward. In other scenarios, the likelihood may have multiple maxima, flat regions, or may approach its supremum without attaining it within the parameter space.
Identifiability refers to whether different parameter values imply distinguishable distributions for the data. If the model is not identifiable, likelihood-based optimization can yield ambiguous solutions because different \(\theta\) values produce the same fitted distribution, leading to identical likelihood (or nearly so).
3.3 Sufficient Statistics (Overview)
A sufficient statistic is a function of the data that retains all information relevant to the parameter for a specific model. When such a statistic exists, the likelihood can often be expressed in a factorized form where the dependence on \(\theta\) occurs only through that statistic.
At a high level, sufficient statistics streamline both theoretical results and practical computation, since they reduce the dimensionality of the input needed for likelihood evaluation and related inference.
3.4 Concavity/Shape in Common Models
The shape of the log-likelihood function influences optimization and uncertainty quantification. In some models, the log-likelihood is concave in the parameter, making the maximum more stable and easier to find with standard methods. In other cases, the surface may be non-concave, leading to local maxima and sensitivity to starting values.
Even when concavity is not guaranteed globally, many practical models exhibit regimes where the likelihood behaves smoothly around the optimum, facilitating numerical fitting.
4 Likelihood in Statistical Inference
4.1 Maximum Likelihood Estimation (MLE)
4.1.1 MLE Mechanics and Optimization
Maximum likelihood estimation selects parameter values that maximize the likelihood (or, equivalently, maximize the log-likelihood). For independent observations, \[ \hat{\theta}_{\mathrm{MLE}}=\arg\max_{\theta} \prod_{i=1}^n f(x_i\mid \theta) =\arg\max_{\theta} \sum_{i=1}^n \log f(x_i\mid \theta). \] When closed-form solutions exist, MLE can be computed directly. More often, optimization relies on numerical methods such as gradient ascent, Newton-type methods, or expectation-maximization variants for latent-variable models.
The mechanics depend heavily on the model: derivative availability, constraints on parameters, and the presence of latent structure all determine which algorithm is appropriate.
4.1.2 Bias and Consistency (Conceptual)
Conceptually, MLE is valued because under broad regularity conditions it tends to concentrate near the true parameter as sample size increases. This property is often summarized as consistency: the estimator converges to the data-generating parameter in large samples.
Finite-sample behavior may show bias, and assessing it typically requires additional analysis or simulation. The core message is that maximizing likelihood targets parameter values that best explain the observed data under the model, while asymptotic guarantees depend on model conditions.
4.2 Likelihood Ratios
4.2.1 Interpretation of Likelihood Ratios
A likelihood ratio compares how well two parameter values explain the same observed data: \[ \frac{L(\theta_1\mid x)}{L(\theta_0\mid x)}. \] Values greater than one favor \(\theta_1\) over \(\theta_0\) in terms of fit to the data. Because likelihood is defined up to multiplicative constants independent of \(\theta\), ratios provide an interpretation that does not depend on such scaling.
Likelihood ratios are widely used in hypothesis testing and model comparison, though the formal link to tests typically involves additional statistical theory.
4.2.2 Relation to Tests (General Idea)
Many test statistics are derived from likelihood ratios. In broad terms, a hypothesis test contrasts a null model or parameter setting against alternatives by measuring how much more likely the data are under the alternative.
While the exact distribution of a likelihood-ratio statistic depends on regularity conditions and the model structure, the qualitative principle remains: large departures in likelihood indicate evidence against the null.
4.3 Bayesian Connection via Bayes’ Rule
4.3.1 Likelihood as the Data-Driven Factor
In Bayesian inference, the posterior distribution combines prior beliefs about parameters with the evidence from data. Bayes’ rule states \[ p(\theta\mid x) \propto p(x\mid \theta)\,p(\theta). \] The term \(p(x\mid \theta)\) is the likelihood. Thus, in Bayesian updating, likelihood acts as the data-driven factor that reweights the prior toward parameter values that better predict the observed outcomes.
Although likelihood and posterior both depend on the same model ingredients, their meanings differ: likelihood compares parameter values given data, while the posterior represents a normalized distribution over parameter values reflecting uncertainty.
5 Practical Modeling Workflows
5.1 Choosing a Statistical Model
Likelihood-based workflows assume a specified probabilistic model for the data. Model choice influences the form of the likelihood and, consequently, the fitted parameters. Selecting an appropriate model typically involves considering data type (binary counts versus continuous measurements), the expected noise structure, and any domain-motivated features such as heterogeneity or correlation.
In practice, model choice often includes iterative refinement: propose a model, fit it via likelihood methods, then check whether its implied behavior resembles observed patterns.
5.2 Computing Likelihoods for Common Distributions
5.2.1 Bernoulli/Binomial Models
For Bernoulli data, where observations are \(0/1\) with success probability \(p\), the likelihood for a single observation \(x\) is \[ L(p\mid x)=p^x(1-p)^{1-x}. \] For \(n\) independent Bernoulli trials with \(k\) successes, the binomial likelihood is \[ L(p\mid k)=\binom{n}{k}p^k(1-p)^{n-k}. \] Since the binomial coefficient does not depend on \(p\), it often drops out in MLE optimization, leaving a simpler expression proportional to \(p^k(1-p)^{n-k}\).
5.2.2 Gaussian Models (Normal Likelihood)
For normal observations with mean \(\mu\) and variance \(\sigma^2\), the likelihood uses the normal density: \[ L(\mu,\sigma^2\mid x)=\prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i-\mu)^2}{2\sigma^2}\right). \] Taking logs yields a sum of squared residual terms, making Gaussian likelihood central to least-squares connections. In many settings, maximizing the normal likelihood corresponds closely to minimizing squared errors, subject to how parameters like \(\sigma^2\) are handled.
5.2.3 Poisson Models
For count data modeled as Poisson with rate \(\lambda\), the likelihood for a single observation \(x\) is \[ L(\lambda\mid x)=\frac{\lambda^x e^{-\lambda}}{x!}. \] For independent counts \(x_1,\dots,x_n\), \[ L(\lambda)=\prod_{i=1}^n \frac{\lambda^{x_i} e^{-\lambda}}{x_i!} \propto \lambda^{\sum x_i} e^{-n\lambda}. \] Again, factors not involving \(\lambda\) can be ignored for optimization, and the log-likelihood becomes a simple expression in \(\log \lambda\) and \(\lambda\).
5.3 Handling Uncertainty and Model Checking
5.3.1 Residual Diagnostics (High-Level)
After fitting a model, analysts often examine residuals or other diagnostic quantities to assess whether assumptions hold. While residuals are most directly motivated by regression contexts, the broader idea is to compare what the model predicts against what the data actually show, focusing on systematic patterns that indicate mis-specification.
In likelihood frameworks, model checking may also involve evaluating predictive distributions, out-of-sample performance, or calibration of uncertainty estimates, depending on the inference approach.
5.3.2 Overfitting Considerations
Likelihood optimization can improve fit to the observed sample, including by choosing overly flexible parameterizations that do not generalize. Overfitting is more likely with complex models, small datasets, or when regularization is absent.
To mitigate it, practitioners often use techniques such as cross-validation, information criteria, or explicit regularization priors in Bayesian settings. The overarching principle is to balance improved likelihood with evidence that the model captures stable structure rather than noise.
6 Common Misconceptions
6.1 “Likelihood Is a Probability” Myth
A frequent confusion is treating likelihood values as probabilities of parameter values. Although likelihood and probability share similar formulas in many cases, likelihood is not a probability distribution over \(\theta\) by default. It does not automatically sum or integrate to one across the parameter space, and its absolute scale lacks direct probabilistic interpretation.
Likelihood is primarily a tool for comparing parameter values given observed data.
6.2 Confusing Parameter Likelihood with Posterior Probability
Another misconception mixes up the likelihood function with the posterior distribution from Bayesian inference. The posterior does represent uncertainty about parameters and is normalized. The likelihood alone reflects compatibility with the data but does not incorporate prior beliefs or provide a normalized uncertainty statement.
In Bayesian workflows, the likelihood is multiplied by the prior and then normalized to produce the posterior; interpreting the likelihood as the posterior skips these steps.
6.3 Comparing Likelihoods Across Incompatible Models
Likelihood comparisons are meaningful only when the compared quantities are defined under compatible modeling frameworks. Comparing likelihood values from different model classes, or from models with different observation structures, can be misleading because the likelihood incorporates model-specific normalization and assumptions.
For model comparison, approaches such as likelihood-ratio tests (within appropriate settings) or information criteria are typically used to ensure the comparison accounts for differences in model complexity and structure.
7 Extended Concepts (Optional Survey)
7.1 Profile Likelihood (Idea)
Profile likelihood reduces dimensionality by focusing on parameters of interest while optimizing out nuisance parameters. If \(\theta=(\psi,\lambda)\) with \(\psi\) the parameter of interest and \(\lambda\) nuisance, the profile likelihood is \[ \mathrm{PL}(\psi)=\sup_{\lambda} L(\psi,\lambda). \] This yields a one-dimensional (or lower-dimensional) function useful for constructing confidence sets or studying sensitivity when nuisance parameters complicate direct analysis.
7.2 Partial Likelihood (Idea)
Partial likelihood is used when the full likelihood is difficult to compute or when only certain components of the model are informative for parameter estimation. In some settings, likelihood contributions can be separated so that parameters can be estimated using a reduced objective that preserves relevant information while avoiding intractable terms.
Though its details depend on the model class, the general goal is computational or inferential tractability.
7.3 Pseudolikelihood in Structured Models (Overview)
Pseudolikelihood is an approximation method for complex structured models, where the full likelihood involves difficult dependencies. Instead of modeling the joint distribution directly, pseudolikelihood replaces it with a product of local conditional distributions.
This approximation can be practical in high-dimensional settings or graphical models, offering a tractable criterion for parameter fitting. Accuracy depends on how well the local conditional structure captures the key dependence patterns of the data.