1 Definition and basic idea

Maximum likelihood estimation is a general procedure for inferring unknown parameters in a statistical model. It selects the parameter values under which the observed data would be most probable, or equivalently, would receive the highest likelihood. Because it turns estimation into an optimization problem, the approach is widely used in both theoretical and applied statistics.

1.1 Statistical model and parameters

A statistical model specifies a family of probability distributions indexed by one or more parameters. These parameters control features such as location, spread, rate, or shape. In practice, the model is chosen to represent the data-generating process approximately, and the parameters summarize the unknown characteristics to be estimated.

1.2 Likelihood function

The likelihood function is obtained by treating the observed data as fixed and viewing the model parameters as variable. It is proportional to the probability of the observed sample under the assumed model. Although it has the same algebraic form as a probability function, it is interpreted differently: as a measure of how well each parameter value explains the data.

1.3 Maximum likelihood principle

The maximum likelihood principle states that the preferred estimate is the parameter value that maximizes the likelihood function. Intuitively, it identifies the model setting that best matches the observed sample according to the assumed probabilistic structure. This principle is attractive because it is simple, broadly applicable, and often leads to estimators with strong long-run performance.

1.4 Maximum likelihood estimator versus maximum likelihood estimate

The maximum likelihood estimator is the rule or formula that maps data to parameter values. The maximum likelihood estimate is the particular numerical value obtained after the data are observed. In many texts, the terms are used interchangeably, but the distinction is between the estimating procedure and its realized outcome.

2 Formulation

Maximum likelihood can be formulated for discrete and continuous data, as well as for samples with multiple observations. The exact expression depends on the type of model, but the underlying aim remains the same: to find parameter values that maximize the likelihood of the data.

2.1 Discrete data

For discrete outcomes, the likelihood is based on the probability mass function of the model. Each observed value contributes a probability, and the joint likelihood is built from these terms. The parameter values are then compared by how much probability they assign to the realized sample.

2.2 Continuous data

For continuous outcomes, the likelihood uses the probability density function. Since the probability of any exact value is zero, the density provides the relevant measure of relative support for different parameter values. The larger the density at the observed points, the more favorable the parameter choice under the model.

2.3 Joint likelihood for independent observations

When observations are independent, the joint likelihood is the product of the individual likelihood contributions. This factorization greatly simplifies many calculations and is one reason maximum likelihood methods are so practical. In repeated-sample settings, it also makes clear how evidence accumulates across data points.

2.4 Log-likelihood function

The log-likelihood is the natural logarithm of the likelihood function. It is used because it converts products into sums, which are easier to differentiate and compute. Maximizing the log-likelihood is equivalent to maximizing the likelihood itself, but the former is usually more convenient for analysis and computation.

3 Methods of finding the estimator

Finding a maximum likelihood estimator often requires solving an optimization problem. In some cases a closed-form solution is available, while in others iterative numerical methods are needed. The choice of method depends on the model structure and the complexity of the likelihood surface.

3.1 Analytical solutions

Some models yield explicit formulas for the maximizing parameter values. These solutions are especially common in simple distributions and standard linear models. Analytical derivations are valuable because they provide insight into how the estimator depends on the data.

3.2 Numerical optimization

Many likelihood functions cannot be maximized exactly by algebraic manipulation. In such cases, algorithms such as gradient ascent, Newton-type methods, and expectation-maximization are used to search for the optimum. Numerical optimization is central to modern likelihood-based modeling.

3.3 Derivatives and score equations

The score is the derivative of the log-likelihood with respect to the parameters. Setting the score equal to zero yields the score equations, which characterize candidate maxima. Second derivatives or related curvature information are then used to check whether the solution is a maximum and to assess its stability.

3.4 Constrained maximization

Some estimation problems impose restrictions on the parameters, such as nonnegativity, ordering, or normalization constraints. Constrained maximization incorporates these conditions directly into the optimization task. Methods such as Lagrange multipliers or specialized constrained algorithms are often used in these settings.

4 Examples

Simple distributions provide standard illustrations of maximum likelihood estimation. These examples show how the general principle produces familiar estimators and how the likelihood reflects the structure of the data.

4.1 Bernoulli distribution

For Bernoulli data, the parameter represents the probability of success in a sequence of binary outcomes. The likelihood is maximized by the sample proportion of successes. This result makes the Bernoulli case one of the clearest examples of maximum likelihood estimation.

4.2 Binomial distribution

In the binomial setting, the parameter again represents a success probability, but the data are typically summarized as the number of successes in a fixed number of trials. The maximum likelihood estimate is the observed proportion of successes. This estimate is natural because it matches the empirical frequency of the event.

4.3 Normal distribution

For a normal distribution, maximum likelihood estimation is commonly used for the mean and variance. The estimator for the mean is the sample average, while the variance estimate is based on the average squared deviation from that mean. These formulas illustrate how likelihood methods recover standard summary statistics.

4.4 Poisson distribution

For Poisson data, the parameter describes the event rate or expected count over a fixed interval. The maximum likelihood estimate is the sample mean of the counts. This reflects the fact that the observed average count provides the strongest support for the underlying rate.

5 Properties

Maximum likelihood estimators have several important large-sample properties. These properties help explain why the method is so widely trusted in statistical inference, even when exact finite-sample results are difficult to obtain.

5.1 Consistency

A consistent estimator converges to the true parameter value as the sample size grows. Under suitable conditions, maximum likelihood estimators are consistent. This means that with enough data, the estimator tends to recover the parameter that generated the sample.

5.2 Asymptotic normality

In large samples, maximum likelihood estimators are often approximately normally distributed around the true parameter value. This approximation supports standard errors, confidence intervals, and hypothesis tests. It is one of the main reasons likelihood methods are so useful in practice.

5.3 Efficiency

Efficiency refers to low variance among unbiased or approximately unbiased estimators. Under regular conditions, maximum likelihood estimators are asymptotically efficient, meaning they achieve the lowest possible asymptotic variance among a broad class of estimators. This property contributes to their strong performance in large samples.

5.4 Invariance property

The invariance property states that if a parameter is estimated by maximum likelihood, then any smooth function of that parameter is estimated by applying the same function to the maximum likelihood estimate. This makes the method especially convenient for derived quantities such as odds ratios, variances, or transformed scales.

5.5 Bias and finite-sample behavior

Although maximum likelihood estimators are often attractive asymptotically, they can exhibit bias in small samples. Finite-sample behavior depends on the model, sample size, and degree of curvature in the likelihood. As a result, practical work sometimes uses bias corrections or alternative estimators when small-sample accuracy is important.

6 Theoretical foundations

The theory behind maximum likelihood estimation connects optimization, information theory, and asymptotic statistics. Several core concepts explain when the method works well and how its precision can be quantified.

6.1 Regularity conditions

Regularity conditions are technical assumptions that ensure standard likelihood results hold. They typically involve smoothness, differentiability, identifiability, and the ability to interchange differentiation and expectation. These conditions are important for proving consistency, asymptotic normality, and related results.

6.2 Fisher information

Fisher information measures the amount of information that the observed data carry about a parameter. It is derived from the curvature of the log-likelihood near its maximum and is closely tied to estimation precision. Larger information generally corresponds to smaller uncertainty in the estimator.

6.3 Cramér–Rao lower bound

The Cramér–Rao lower bound gives a theoretical lower limit on the variance of unbiased estimators under suitable conditions. It provides a benchmark for judging estimator performance. Maximum likelihood estimators often approach this bound asymptotically, which helps explain their efficiency.

6.4 Asymptotic theory

Asymptotic theory studies the behavior of estimators as sample size becomes large. For maximum likelihood, this theory shows how the estimator concentrates near the true value and how its distribution can be approximated by a normal law. It also supports practical tools such as Wald intervals and likelihood-based tests.

7 Connections to other estimation methods

Maximum likelihood estimation is closely related to many other inferential approaches. Some methods resemble it algebraically, while others use different principles but produce similar results under certain conditions.

7.1 Method of moments

The method of moments estimates parameters by matching sample moments to theoretical moments. It is often simpler to compute than maximum likelihood, though usually less efficient. In some models, the two methods give similar answers, while in others they differ substantially.

7.2 Bayesian estimation

Bayesian estimation combines prior information with the likelihood to form a posterior distribution. Maximum likelihood uses only the likelihood and ignores prior beliefs. Nevertheless, likelihood functions are central in Bayesian analysis because they provide the data-based component of posterior updating.

7.3 Least squares estimation

Least squares estimation chooses parameters to minimize the sum of squared residuals. In many common models, especially those with normally distributed errors, least squares and maximum likelihood coincide or are closely connected. This relationship explains why least squares occupies such a central role in statistical modeling.

7.4 Maximum a posteriori estimation

Maximum a posteriori estimation selects the parameter value with the highest posterior probability or density. It differs from maximum likelihood by incorporating a prior distribution. When the prior is flat or weakly informative, the two approaches may yield similar estimates.

8 Practical considerations

Although maximum likelihood is conceptually straightforward, real applications often involve complications. The quality of the estimate depends not only on the model but also on numerical, structural, and diagnostic issues.

8.1 Identifiability

A model is identifiable when different parameter values produce different data distributions. If identifiability fails, the likelihood may not single out a unique maximum. Ensuring identifiability is therefore essential for meaningful estimation.

8.2 Multiple maxima

Some likelihood functions have more than one local maximum. This can make optimization difficult, since an algorithm may stop at a suboptimal solution. Careful initialization and diagnostic checks are often needed to reduce this risk.

8.3 Numerical instability

Likelihood calculations can suffer from overflow, underflow, or loss of precision, especially for large samples or highly concentrated distributions. Using log-likelihoods, stable recursions, and appropriate scaling can improve numerical reliability. Stable computation is often as important as the theoretical formulation.

8.4 Model misspecification

Maximum likelihood assumes that the chosen model is a reasonable representation of the data-generating process. When the model is misspecified, the estimator may still be useful, but its interpretation changes. In such cases, it may target the best-fitting parameter within the incorrect model rather than the true generating mechanism.

8.5 Computational software

Statistical software packages implement maximum likelihood estimation for many models. These tools automate optimization, standard error computation, and diagnostic output. Even so, users must still assess convergence, check assumptions, and interpret results carefully.

Many important variants of maximum likelihood have been developed to handle complex data, dependence structures, or additional modeling goals. These extensions preserve the core idea while adapting it to more specialized settings.

9.1 Penalized maximum likelihood

Penalized maximum likelihood adds a penalty term to the log-likelihood to discourage overly complex estimates. This approach is useful for regularization, shrinkage, and preventing overfitting. Common penalties include ridge and lasso-type forms.

9.2 Pseudo-likelihood

Pseudo-likelihood replaces the full likelihood with a simpler approximation that is easier to compute. It is especially useful in models with complicated dependence structures. Although it may be less exact than full maximum likelihood, it can still provide effective estimates.

9.3 Partial likelihood

Partial likelihood focuses on a subset of the full likelihood that is sufficient for estimating certain parameters of interest. It is widely used when nuisance parameters are difficult to model directly. This strategy preserves key inferential information while simplifying computation.

9.4 Robust maximum likelihood

Robust maximum likelihood modifies standard procedures to reduce sensitivity to departures from ideal model assumptions. It may involve adjusted standard errors, alternative objective functions, or methods designed to tolerate outliers and misspecification. Such approaches aim to retain much of the efficiency of likelihood methods while improving practical resilience.

10 Applications

Maximum likelihood estimation appears in a wide range of statistical models and analytical workflows. Its flexibility makes it a standard tool in both classical statistics and modern data analysis.

10.1 Regression models

In regression analysis, maximum likelihood is used to estimate coefficients linking predictors to an outcome. Depending on the response type, the model may be linear, logistic, or count-based. The method provides a unified framework for fitting many regression specifications.

10.2 Generalized linear models

Generalized linear models extend regression to nonnormal response distributions. Maximum likelihood supplies parameter estimates for the linear predictor and distributional components. This framework underlies many commonly used models for binary, count, and categorical data.

10.3 Time series models

Time series models often rely on likelihood methods to estimate parameters governing serial dependence and variability. Maximum likelihood can be adapted to autoregressive, moving-average, and state-space formulations. It is especially valuable when the dependence structure is explicit and the model is well specified.

10.4 Survival analysis

In survival analysis, likelihood methods are used to estimate event-time distributions and hazard-related parameters. They are particularly helpful when data include censoring, which limits the observed information. Maximum likelihood provides a principled way to incorporate incomplete time-to-event observations.

10.5 Machine learning applications

Many machine learning algorithms are built around likelihood maximization or closely related objective functions. Examples include probabilistic classification, latent variable models, and neural network training under probabilistic loss functions. In these settings, the maximum likelihood perspective helps connect prediction, inference, and optimization.