1 Foundations

Maximum likelihood is a general framework for estimating unknown model parameters from observed data. The central idea is to choose the parameter values under which the data would be most probable, according to the assumed statistical model. In practice, this makes the method applicable to a wide range of settings, from simple probability distributions to complex multivariate systems.

1.1 Statistical inference

Maximum likelihood belongs to the broader field of statistical inference, which uses data to draw conclusions about an underlying process or population. Inference often asks how likely different parameter values are, how uncertain an estimate may be, and whether competing models can explain the observed sample. Maximum likelihood addresses these questions by translating them into an optimization problem.

1.2 Likelihood function

The likelihood function expresses the plausibility of the observed data as a function of the parameters. Although it is built from a probability model, it is treated differently from a probability distribution: the data are fixed, while the parameters vary. The shape of the likelihood function indicates which parameter values are supported most strongly by the sample.

1.3 Parameter estimation

Parameter estimation aims to find numerical values that summarize or calibrate a model. Maximum likelihood estimation selects the values that maximize the likelihood function. This approach is attractive because it can be applied to many models without requiring special ad hoc formulas for each case.

1.4 Comparison with other estimation methods

Maximum likelihood is often compared with methods such as method of moments, least squares, and Bayesian estimation. Compared with moment-based approaches, it usually uses the full probability model and can be more efficient when that model is appropriate. Compared with least squares, it is more general, since least squares appears as a special case for certain error distributions. In relation to Bayesian methods, maximum likelihood produces a point estimate without using a prior distribution.

2 Historical development

The method developed gradually from early work in probability and observational science. Its modern form emerged through the formalization of likelihood as a distinct inferential concept and through advances in asymptotic theory and computation.

2.1 Early statistical origins

Early statistical analysis often focused on fitting curves, summarizing measurements, and selecting parameters that matched observed frequencies. These ideas anticipated later likelihood methods, even when the terminology was not yet established. The growth of probability theory in the nineteenth century created the mathematical setting needed for a more systematic approach.

2.2 Fisher’s contributions

Ronald A. Fisher played a major role in defining likelihood as a principle of inference. He distinguished likelihood from probability and used it to build a unified theory of estimation and testing. His work also helped establish key ideas such as efficient estimation, sufficiency, and asymptotic behavior.

2.3 Growth in modern statistics

During the twentieth century, maximum likelihood became a central tool in mathematical statistics. Its appeal increased as researchers developed general results on large-sample properties and as numerical methods made difficult optimization problems tractable. The method later spread into applied fields where flexible model fitting was essential.

3 Mathematical formulation

Maximum likelihood rests on a probability model for the data and a parameter space describing the possible values of the unknown quantities. The method then identifies the parameter value that maximizes the likelihood, or equivalently the log-likelihood.

3.1 Probability models

A probability model specifies how data are generated under different parameter values. The data may be independent observations, dependent time-ordered measurements, or outcomes from a multivariate system. The model can be discrete or continuous, provided it assigns probabilities or densities in a coherent way.

3.2 Likelihood and log-likelihood

For observed data, the likelihood is the model expression viewed as a function of the parameters. Because products of many probabilities or densities can be numerically awkward, analysts often work with the log-likelihood. Taking logarithms preserves the location of the maximum while simplifying differentiation and computation.

3.3 Optimization problem

The estimation problem is to find the parameter value that yields the highest likelihood. This may involve setting derivatives to zero, checking second-order conditions, or using numerical search methods when no closed-form solution exists. The optimization can be straightforward for simple models and highly demanding for complex ones.

3.4 Constraints and parameter spaces

Parameters often must satisfy restrictions such as positivity, boundedness, or sum-to-one conditions. These constraints define the allowable parameter space and affect both the existence and location of the maximum. In some models, the optimum occurs on the boundary rather than in the interior, which requires careful interpretation.

4 Maximum likelihood estimation

Maximum likelihood estimation, often abbreviated MLE, produces a point estimate by maximizing the likelihood function over the parameter space. The resulting estimator is widely used because it is principled, broadly applicable, and often performs well in large samples.

4.1 Point estimation

An MLE is a single set of parameter values chosen as the best fit under the model. It summarizes the sample in a compact form and can be used for prediction, classification, or further inference. When several parameters are estimated jointly, the likelihood is maximized over all of them at once.

4.2 Closed-form solutions

Some models yield explicit formulas for the maximum likelihood estimator. In such cases, the estimate can be written directly in terms of the data, without iterative calculation. Classic examples include the mean of a normal distribution with known variance and the rate parameter of certain exponential-family models.

4.3 Numerical optimization methods

When a closed-form answer is unavailable, numerical optimization is used. These methods search for the maximum by repeatedly improving a provisional parameter guess. Their performance depends on the smoothness of the likelihood surface, the quality of starting values, and the presence of constraints.

4.3.1 Gradient-based methods

Gradient-based methods use first derivatives to indicate the direction of increase. By moving along the gradient or its variants, the algorithm can approach a local or global maximum. These methods are popular because they scale well to high-dimensional problems.

Newton–Raphson methods use both first and second derivatives to refine parameter estimates. By incorporating curvature information, they often converge rapidly near the optimum. Related procedures include Fisher scoring and quasi-Newton methods, which trade exact curvature calculations for greater stability or efficiency.

4.3.3 Iterative procedures

Many likelihood problems are solved by iteration rather than direct formulas. The algorithm begins with an initial guess, updates the parameters according to a rule, and stops when changes become sufficiently small. Convergence criteria may involve the parameter values, the log-likelihood, or the gradient norm.

4.4 Examples of estimators

Common maximum likelihood estimators include the sample mean for a normal location model, the sample proportion for Bernoulli data, and the rate estimate in Poisson and exponential models. In more elaborate settings, MLEs can estimate regression coefficients, transition probabilities, or shape parameters in multivariate distributions.

5 Statistical properties

Maximum likelihood estimators are valued not only for their practical usefulness but also for their favorable theoretical behavior under broad conditions. These properties usually become clearest in large samples.

5.1 Consistency

Consistency means that the estimator converges to the true parameter value as sample size increases. Under suitable regularity conditions, maximum likelihood estimators are consistent, so they become more accurate as more data are observed. This feature is one of the main reasons for the method’s wide acceptance.

5.2 Efficiency

Efficiency refers to achieving low variance among unbiased or approximately unbiased estimators. In many standard models, maximum likelihood estimators attain or approach the smallest possible asymptotic variance. This makes them attractive when the model is correctly specified and the sample is sufficiently large.

5.3 Asymptotic normality

Asymptotic normality means that, after suitable scaling, the estimator approaches a normal distribution as sample size grows. This result supports approximate standard errors and interval estimates. It also provides a foundation for many common inferential procedures based on the observed information matrix.

5.4 Bias and finite-sample behavior

In finite samples, maximum likelihood estimators may be biased, especially in small or irregular samples. The direction and size of the bias depend on the model and the parameter being estimated. Although asymptotic theory gives valuable guidance, actual performance can differ noticeably when data are limited.

5.5 Invariance property

The invariance property states that if a parameter is estimated by maximum likelihood, then any smooth transformation of that parameter is estimated by applying the same transformation to the MLE. This simplifies inference for derived quantities such as variances, odds ratios, or predicted intensities.

6 Inference based on maximum likelihood

Maximum likelihood supports more than point estimation. It also underlies methods for quantifying uncertainty, testing hypotheses, and comparing competing models.

6.1 Confidence intervals

Confidence intervals based on likelihood can be derived from curvature, likelihood-ratio behavior, or asymptotic normal approximations. These intervals aim to describe a plausible range for the parameter rather than a single best value. Their quality depends on sample size and the accuracy of the underlying approximation.

6.2 Hypothesis testing

Likelihood-based hypothesis tests evaluate whether a restricted model fits substantially worse than a more general one. These tests are widely used because they connect naturally to model fit and can often be derived from the same likelihood surface used for estimation.

6.2.1 Likelihood ratio tests

Likelihood ratio tests compare the maximum likelihood under a null model with the maximum under an alternative model. A large difference suggests that the more flexible model captures the data better. This test has a strong theoretical basis and is common in many branches of statistics.

6.2.2 Wald tests

Wald tests assess whether an estimated parameter differs significantly from a hypothesized value. They use the estimate and its estimated standard error to form a test statistic. Wald procedures are convenient, though they may be less reliable than alternatives in some nonlinear or small-sample settings.

6.2.3 Score tests

Score tests evaluate the slope of the likelihood at the null hypothesis without requiring full estimation under the alternative. They are useful when the alternative model is difficult to fit directly. In some problems, they provide a computationally economical route to significance testing.

6.3 Model comparison

Likelihood values can be used to compare nested or competing models. Information criteria and likelihood-based tests help balance goodness of fit against model complexity. Such comparisons are especially useful when several plausible specifications can explain the data.

7 Applications

Maximum likelihood is used in many fields because it adapts to diverse kinds of data and model structures. Its flexibility makes it a standard method in both theoretical and applied work.

7.1 Generalized linear models

In generalized linear models, maximum likelihood estimates regression coefficients for outcomes such as counts, binary responses, or categorical data. The method provides a coherent way to fit models with non-normal error structures and non-linear link functions. It is central to logistic and Poisson regression.

7.2 Time series analysis

Time series models often rely on likelihood to estimate autocorrelation, trend, and volatility parameters. The method can accommodate dependence across observations, which is essential when data are ordered in time. It is widely used in autoregressive, moving-average, and state-space settings.

7.3 Survival analysis

Survival analysis studies time-to-event data, often with censoring. Maximum likelihood helps estimate hazard-related quantities and model covariate effects. It is especially valuable when event times are incompletely observed or when the risk process varies over time.

7.4 Machine learning

Many machine learning models are trained by maximizing a likelihood or an equivalent objective. Examples include probabilistic classifiers, mixture models, and latent-variable methods. In this setting, maximum likelihood often serves as a bridge between statistical modeling and optimization-based learning.

7.5 Biostatistics and epidemiology

In biostatistics and epidemiology, maximum likelihood is used for regression, disease-rate modeling, and the analysis of clinical and observational data. The method supports estimation under incomplete observation, grouped data, and complex sampling designs. Its ability to quantify uncertainty makes it especially useful in scientific studies.

8 Computational aspects

Practical maximum likelihood analysis depends heavily on computation. Even when the statistical theory is well understood, numerical implementation may be challenging.

8.1 Numerical stability

Likelihood calculations can involve very small probabilities, large products, or poorly scaled variables. Stable computation often requires logarithms, careful algebraic rearrangement, and attention to overflow or underflow. These precautions can substantially improve reliability.

8.2 Convergence issues

Iterative algorithms may converge slowly, stop prematurely, or fail to converge at all. Problems often arise from weak starting values, flat likelihood surfaces, or extreme parameter correlations. Monitoring the optimization process is therefore an important part of estimation.

8.3 Local maxima and identifiability

Some likelihood functions have more than one local maximum, which can lead to different solutions depending on initialization. Identifiability is also crucial: if distinct parameter values produce the same model distribution, the maximum may not be unique. Both issues complicate interpretation and can reduce confidence in the estimate.

8.4 Software implementations

Modern statistical software provides routines for maximum likelihood estimation across many model classes. These implementations typically combine optimization algorithms, derivative calculations, and convergence checks. Users still need to verify assumptions, inspect diagnostics, and interpret outputs with care.

Maximum likelihood has inspired several important extensions and closely related estimation strategies. These methods preserve the basic idea of optimizing a criterion while adapting it to more complex problems.

9.1 Penalized likelihood

Penalized likelihood adds a penalty term to the ordinary likelihood to discourage overly complex models or unstable estimates. This approach is common in high-dimensional settings and in situations where regularization improves prediction. It trades a small amount of bias for reduced variance and greater robustness.

9.2 Bayesian estimation

Bayesian estimation combines a likelihood with a prior distribution to produce a posterior distribution over parameters. While the goals differ from those of maximum likelihood, the two approaches are closely connected because they use the same data model. Under weak priors and large samples, Bayesian estimates may resemble maximum likelihood estimates.

9.3 Restricted maximum likelihood

Restricted maximum likelihood, often used in mixed-effects models, modifies the usual likelihood to account for nuisance parameters such as variance components. It is designed to improve estimation of these quantities by adjusting for the loss of degrees of freedom caused by fixed-effect estimation. This method is especially useful in hierarchical models.

9.4 Expectation-maximization algorithm

The expectation-maximization algorithm is an iterative technique for likelihood maximization in models with latent or incomplete data. It alternates between estimating hidden quantities and updating the parameters. The method is widely used because it can handle problems that are awkward to optimize directly.

10 Criticisms and limitations

Despite its strengths, maximum likelihood is not universally ideal. Its performance depends on the correctness of the model, the amount of data available, and the feasibility of computation.

10.1 Sensitivity to model misspecification

If the chosen model is substantially wrong, maximum likelihood can produce precise but misleading estimates. The method optimizes fit within the assumed model, not necessarily to the true data-generating process. This makes model checking an essential companion to estimation.

10.2 Small-sample limitations

In small samples, maximum likelihood estimators may have noticeable bias or unstable variance. Asymptotic approximations can be too optimistic when the dataset is limited. Alternative estimators or bias corrections may then be preferable.

10.3 Computational complexity

Some likelihood problems are expensive to solve, especially in high dimensions or with latent variables. The optimization may require many iterations and careful tuning. As models become more elaborate, computation can become the main practical obstacle.

10.4 Non-identifiable models

When a model is non-identifiable, the data do not contain enough information to separate certain parameters. In such cases, maximum likelihood may fail to produce a unique or meaningful solution. Recognizing identifiability problems is therefore essential before relying on the estimate.