Maximum likelihood estimation

1 Fundamentals

Maximum likelihood estimation is a general method for choosing model parameters that best explain observed data under a specified probabilistic framework. The central idea is to treat the observed sample as fixed and to compare how plausible it would be under different parameter values. The parameter value, or set of values, that makes the data most probable is taken as the estimate.

1.1 Statistical model and parameters

A statistical model describes a family of possible data-generating mechanisms. It usually consists of a probability distribution or distributional form indexed by one or more parameters. These parameters may represent a location, spread, rate, shape, or other structural feature of the model. In many settings, some parameters are of direct substantive interest, while others serve as nuisance quantities needed to complete the specification.

1.2 Likelihood function

The likelihood function expresses how plausible the observed data are for each possible parameter value. Although it is built from the same mathematical expression as a probability model, it is interpreted differently: the data are treated as given, and the parameter is the variable being compared across alternatives. This makes the likelihood a tool for estimation rather than for predicting future events directly.

1.2.1 Construction from observed data

To construct a likelihood, one starts with the chosen statistical model and substitutes the observed sample into its probability formula. For independent observations, the likelihood is often a product of individual terms; for dependent data, the joint distribution is used instead. The resulting function is then viewed as a function of the parameters alone.

1.2.2 Distinction from probability density or mass functions

A probability density or mass function assigns probabilities or densities to outcomes when the parameter is fixed. A likelihood function uses the same formula but reverses the roles: the outcome is fixed and the parameter varies. Because of this reversal, likelihood values are not probabilities of parameters, and they do not need to sum or integrate to one.

1.3 Maximum likelihood principle

The maximum likelihood principle selects the parameter value that maximizes the likelihood function. Intuitively, this is the parameter setting under which the observed data appear most compatible with the model. When several parameters are involved, the maximizing set is called the maximum likelihood estimate, or MLE.

1.4 Log-likelihood

The log-likelihood is the natural logarithm of the likelihood function. It is widely used because logarithms convert products into sums, making expressions easier to analyze and compute. Since the logarithm is a strictly increasing function, maximizing the log-likelihood is equivalent to maximizing the original likelihood.

2 Estimation procedure

In practice, maximum likelihood estimation proceeds by specifying the model, writing down the likelihood for the observed sample, and finding the parameter values that maximize it. The resulting procedure can be simple for familiar distributions or highly technical for complex models. The same general logic applies across both elementary and advanced settings.

2.1 Formulating the likelihood

The first step is to express the joint distribution of the observed data in terms of the unknown parameters. This requires choosing an appropriate model and stating any assumptions about independence, identical distribution, or conditional structure. Once the joint expression is written, it becomes the objective function to be optimized.

2.2 Maximizing the likelihood

Maximization can be carried out by calculus, algebra, or numerical algorithms depending on the form of the likelihood. In favorable cases, the optimum can be derived exactly. In more complicated models, software routines search iteratively for the best-fitting parameter values.

2.2.1 Analytical solutions

An analytical solution is obtained when the likelihood equations can be solved explicitly. This often occurs in simple models such as Bernoulli, Poisson, or normal distributions with standard assumptions. Analytical estimators are attractive because they are exact, transparent, and easy to interpret.

2.2.2 Numerical optimization

Many real models do not yield closed-form maximizers. In such cases, one uses iterative numerical methods that improve the parameter values step by step. These procedures may rely on gradients, Hessians, or other information about the shape of the likelihood surface.

2.3 Parameter constraints

Parameters are often restricted to a feasible range, such as positive variance or probabilities between zero and one. Constraints can alter the location of the maximum and may place the optimum on the boundary of the parameter space. Careful handling of these restrictions is essential for correct estimation.

2.4 Multiple maxima and identifiability

A likelihood function may have more than one maximizer, especially in symmetric or poorly identified models. Identifiability refers to whether different parameter values produce distinct distributions for the observed data. If a model is not identifiable, the data may not contain enough information to determine a unique estimate.

3 Properties of maximum likelihood estimators

Maximum likelihood estimators are valued not only for their computational convenience but also for their large-sample behavior. Under suitable regularity conditions, they often perform well as sample size grows. These properties help explain their central role in statistical inference.

3.1 Consistency

Consistency means that the estimator converges to the true parameter value as the sample size increases. For MLEs, consistency generally holds when the model is correctly specified and the usual mathematical conditions are satisfied. This makes maximum likelihood a reliable method in large samples.

3.2 Asymptotic normality

Asymptotic normality refers to the fact that, after suitable scaling, the distribution of the estimator approaches a normal distribution for large samples. This result allows approximate standard errors, confidence intervals, and hypothesis tests to be constructed. It is one of the main reasons MLE is so widely used in applied work.

3.3 Efficiency

Efficiency concerns how much statistical uncertainty an estimator has relative to others. Under ideal regularity conditions, MLEs often achieve the smallest possible asymptotic variance among a broad class of estimators. This optimality is one of the classic arguments in favor of likelihood-based methods.

3.4 Invariance property

The invariance property states that if a parameter is estimated by maximum likelihood, then any smooth transformation of that parameter is estimated by applying the same transformation to the MLE. For example, if a mean or rate is estimated first, a derived quantity such as its reciprocal or logarithm can be obtained directly from the original estimate. This makes MLE especially convenient for derived quantities.

3.5 Bias in finite samples

Although MLEs are often excellent asymptotically, they may be biased in small samples. The direction and size of the bias depend on the model and sample size. In finite data settings, researchers sometimes use bias correction or alternative estimators when improved small-sample performance is important.

4 Common examples

Many familiar statistical models have simple maximum likelihood estimates that illustrate the method clearly. These examples show how likelihood-based reasoning leads to standard estimators used throughout applied statistics. They also provide useful intuition for more complex models.

4.1 Bernoulli and binomial models

For Bernoulli trials, the likelihood depends on the probability of success. The MLE of that probability is the observed proportion of successes. In the binomial setting, the same idea applies, with the estimate given by the number of successes divided by the number of trials.

4.2 Normal distribution

The normal model is a classic example because its likelihood is tractable and its parameters have clear interpretations. Depending on what is assumed known or unknown, the resulting estimators take different forms. The model is also central in theory because of its frequent appearance in asymptotic results.

4.2.1 Unknown mean

When the variance is treated as known, the MLE for the mean is the sample average. This estimate reflects the balance point of the observed data under the normal model. It is one of the simplest and most widely recognized likelihood estimators.

4.2.2 Unknown variance

When the mean is known or simultaneously estimated, the MLE for the variance is based on the average squared deviation from the fitted mean. In the common case where the mean is also estimated, the MLE divides by the sample size rather than by sample size minus one. This distinguishes it from the unbiased sample variance.

4.3 Poisson distribution

For count data modeled by a Poisson distribution, the likelihood leads to an estimate equal to the sample mean. This reflects the model’s property that its mean and variance are both governed by the same rate parameter. Poisson MLEs are frequently used for event counts and occurrence rates.

4.4 Exponential distribution

In the exponential model, which is often used for waiting times, the MLE for the rate parameter is the reciprocal of the sample mean. This result follows from the model’s memoryless structure and its simple likelihood form. It provides a basic example of estimating a positive-scale parameter.

4.5 Linear regression models

In linear regression with normal errors, maximum likelihood coincides with ordinary least squares for the regression coefficients. The likelihood framework also estimates the error variance and provides a basis for testing and interval estimation. This connection helps unify classical regression with broader likelihood theory.

5 Inference based on likelihood

Likelihood methods support not only point estimation but also statistical inference about parameters. By comparing likelihood values under different hypotheses, one can build tests and intervals with strong theoretical foundations. These methods are widely used because they fit naturally with the estimation framework.

5.1 Likelihood ratio tests

A likelihood ratio test compares the maximum likelihood under a null hypothesis with the maximum likelihood under a more general alternative. If the null hypothesis fits the data substantially worse, the test statistic becomes large. In many regular settings, its distribution is well approximated by a chi-square law.

5.2 Wald tests

Wald tests assess whether an estimated parameter is far from a hypothesized value relative to its standard error. They use the estimate and its estimated variance rather than comparing two fitted models directly. These tests are simple to compute, though they can be less stable than some alternatives in difficult problems.

5.3 Score tests

Score tests evaluate whether the slope of the likelihood at the null hypothesis indicates a departure in the direction of an alternative. They require fitting only the model under the null, which can make them computationally efficient. They are also known as Lagrange multiplier tests in some contexts.

5.4 Confidence intervals

Likelihood-based confidence intervals describe parameter values that remain reasonably compatible with the observed data. They may be obtained by inverting likelihood ratio tests or by using asymptotic normal approximations. Such intervals often have good performance when the likelihood is well behaved.

5.5 Profile likelihood

Profile likelihood is formed by maximizing the likelihood over nuisance parameters while holding a parameter of interest fixed. This creates a reduced one-dimensional function that summarizes support for the target parameter. It is especially useful when direct marginalization is difficult or when the parameter of interest is embedded in a larger model.

6 Computational methods

As models grow more complicated, direct maximization becomes less feasible and computational algorithms become essential. Modern statistical software typically implements iterative methods designed to locate local or global maxima efficiently. Good computation is often as important as good theory in practical likelihood analysis.

6.1 Gradient-based optimization

Gradient-based methods use the direction of steepest ascent to improve parameter values. They are effective when derivatives can be calculated accurately or approximated reliably. These methods are common because they scale well to large problems.

6.2 Newton-Raphson method

The Newton-Raphson method uses first and second derivatives to update parameters. By incorporating curvature information, it can converge rapidly near the optimum. However, it may require careful starting values and can behave poorly when the likelihood surface is irregular.

6.3 Expectation-maximization algorithm

The expectation-maximization algorithm is used when data are incomplete, latent, or partially observed. It alternates between estimating hidden quantities and maximizing the expected complete-data likelihood. The procedure is widely applied in mixture models, missing-data problems, and latent-variable settings.

6.4 Iterative reweighted least squares

Iterative reweighted least squares solves certain likelihood problems by repeatedly fitting weighted least-squares approximations. It is especially common in generalized linear models. At each step, the weights reflect the current curvature of the likelihood surface.

6.5 Convergence criteria and initialization

Iterative algorithms require rules for deciding when to stop. Common criteria include small changes in the likelihood, parameter values, or gradient size. Initialization also matters, since poor starting values can lead to slow convergence or attraction to an undesirable local maximum.

7 Extensions and related methods

Maximum likelihood has many variants and close relatives that adapt the basic idea to more complex settings. Some add constraints, while others incorporate prior information or penalty terms. These extensions broaden the method’s usefulness across theory and practice.

7.1 Constrained maximum likelihood

Constrained maximum likelihood estimates parameters subject to restrictions such as equality or inequality conditions. These constraints may arise from physical laws, monotonicity requirements, or model structure. The optimization problem becomes one of maximizing the likelihood within an allowable region.

7.2 Penalized likelihood

Penalized likelihood adds a penalty term to the usual objective to discourage overly complex solutions. This is useful for improving stability, handling high-dimensional settings, or inducing sparsity. The penalty creates a trade-off between fit to the data and model simplicity.

7.3 Bayesian comparison

Bayesian methods also use probability models, but they combine the likelihood with prior information about parameters. In Bayesian analysis, the result is a posterior distribution rather than a point estimate alone. Maximum likelihood corresponds to using the likelihood without a prior term and is often viewed as a frequentist counterpart.

7.4 Generalized linear models

Generalized linear models extend linear regression to non-normal response variables through a link function and an exponential-family distribution. Maximum likelihood provides the standard estimation framework for these models. This makes likelihood methods central to classification, count data analysis, and related tasks.

7.5 Semiparametric and nonparametric settings

In semiparametric models, only part of the structure is specified parametrically, while other components are left more flexible. Nonparametric methods relax parametric assumptions even further. Likelihood ideas still play a role, though the optimization and theoretical analysis become more delicate.

8 Applications

Maximum likelihood estimation appears in many scientific and quantitative disciplines. Its broad appeal comes from its flexibility and its ability to connect modeling, estimation, and inference in a single framework. The same principle can be adapted to very different kinds of data.

8.1 Machine learning

In machine learning, maximum likelihood is used to fit probabilistic classifiers, generative models, and latent-variable systems. It often appears as the training objective for models that predict labels, sequences, or distributions over outcomes. Many loss functions used in practice are negative log-likelihoods.

8.2 Econometrics

Econometrics uses likelihood methods to estimate structural and time-series models, especially when error distributions are specified explicitly. Maximum likelihood supports model comparison and hypothesis testing in settings involving economic behavior, panel data, and stochastic processes. It is valued for its flexibility in complex specifications.

8.3 Biostatistics

In biostatistics, likelihood methods are used to analyze clinical, epidemiological, and laboratory data. They are particularly useful for incomplete observations, survival outcomes, and hierarchical designs. The framework is compatible with both individual-level and grouped data analysis.

8.4 Signal processing

Signal processing applications often model observations as noisy measurements of an underlying source. Maximum likelihood helps estimate frequencies, amplitudes, delays, and other signal parameters. It is also used in detection and estimation problems where the data arise from stochastic systems.

8.5 Experimental science

Experimental sciences rely on MLE to interpret measurement error, calibrate instruments, and estimate physical parameters. The method is common in physics, chemistry, astronomy, and engineering because it offers a principled way to infer unknown quantities from noisy observations. It can also support model checking through likelihood-based comparisons.

9 Limitations and assumptions

Although maximum likelihood is powerful, it depends on assumptions that may not hold perfectly in practice. Its performance can degrade when the model is wrong, the data are unusual, or the sample size is limited. Awareness of these limitations is important for sound application.

9.1 Model misspecification

If the chosen model does not reflect the true data-generating process, the MLE may estimate the best approximation within the misspecified family rather than the real parameter. This can affect both interpretation and uncertainty assessment. Robust diagnostics and model checking are therefore important.

9.2 Sensitivity to outliers

Likelihood-based estimators can be influenced by extreme observations, especially in models with unbounded or heavy-tailed sensitivity. A small number of unusual data points may shift the estimate substantially. Robust alternatives or data inspection may be needed in such cases.

9.3 Small-sample issues

In small samples, the likelihood surface may be irregular and asymptotic approximations may be inaccurate. Standard errors and confidence intervals can then be misleading. Specialized corrections or exact methods may provide better finite-sample performance.

9.4 Nonregular problems

Some models violate the regularity conditions needed for the usual MLE theory. Examples include boundary parameters, mixture distributions, and certain latent-variable settings. In these cases, standard normal approximations and chi-square results may fail or require modification.

10 Historical development

Maximum likelihood emerged as a major idea in modern statistics through the effort to ground inference in a systematic and general principle. Its development combined theoretical work, practical computation, and later advances in numerical methods. Over time, it became one of the most influential concepts in statistical estimation.

10.1 Origins in statistical theory

The roots of maximum likelihood lie in early attempts to infer unknown quantities from observational data using probability models. As statistical theory matured, researchers sought methods that could unify estimation across different kinds of problems. Likelihood became a central organizing principle because of its flexibility and intuitive appeal.

10.2 Fisher's contributions

Ronald A. Fisher played a decisive role in formalizing and promoting maximum likelihood as a general method. He developed much of the modern likelihood framework and connected it to efficient estimation and inferential procedures. His work helped establish likelihood as a foundational concept in twentieth-century statistics.

10.3 Modern computational advances

The rise of digital computing transformed maximum likelihood from a mainly analytical technique into a broadly applicable computational tool. Numerical optimization, simulation-based methods, and specialized algorithms made it possible to fit increasingly complex models. These advances expanded the reach of MLE into modern data analysis and machine learning.