Nested model comparison

1 Definition and concept

Nested model comparison is a family of statistical methods for assessing whether adding flexibility to a model—by moving from a simpler model to a larger one obtained by relaxing parameter constraints—yields a quantitatively better explanation of observed data. The procedure explicitly accounts for the fact that the larger model can fit noise in addition to signal.

1.1 What makes models “nested”

Two models are nested when the smaller (reduced) model is a constrained version of the larger (full) model. Concretely, there exists a parameter restriction such that every parameter choice in the reduced model corresponds to at least one parameter choice in the full model, and the reduced model is recovered by fixing one or more additional parameters of the full model.

1.2 Parameter restrictions and special-case structure

Nesting is usually expressed through constraints like setting additional regression coefficients to zero, fixing a variance component to a constant, or enforcing equalities among parameters. This “special-case” relationship creates a directed link between models: the reduced model is not an unrelated alternative, but a subset of the full model’s parameter space.

1.3 Goals: inference vs prediction

Nested comparisons can support different objectives. For inference, the question is whether the extra terms (or parameters) correspond to a real improvement beyond what the simpler model explains. For prediction, the aim is often to determine whether improved fit translates into better out-of-sample performance. Likelihood-ratio tests and information criteria primarily target fit and model adequacy within the modeling framework, while predictive validation typically requires additional steps such as cross-validation.

1.4 When nested comparison is appropriate

Nested model comparison is most appropriate when:

The larger and smaller models truly share a structured parameter restriction.
The data-generating process is plausibly within the modeling family being compared (e.g., linear or generalized linear models with appropriate link).
The analyst can justify the candidate model set and the constraints defining nesting.
Relevant assumptions and regularity conditions are reasonably satisfied, or at least their violations are addressed through suitable approximations or alternative approaches.

2 Statistical foundations

Nested model comparison relies on comparing how well models fit the data while recognizing that the full model has more free parameters. The logic is often framed through likelihood, deviance, and degrees of freedom.

2.1 Likelihood and deviance perspectives

Likelihood provides a natural basis for nested comparisons because the full model’s maximum likelihood estimate will always achieve at least as high a likelihood as the reduced model’s maximum likelihood estimate (within the same data and model form).

2.1.1 Log-likelihood differences

Let \(\ell_F\) be the maximum log-likelihood under the full model and \(\ell_R\) under the reduced model. The difference \(\ell_F-\ell_R\) summarizes the gain in fit. Under suitable conditions and for correctly specified models, the distribution of this gain can be characterized under the reduced model as a reference for testing.

2.1.2 Residual sum of squares for common models

For ordinary least squares linear regression with normally distributed errors and equal variance, maximum likelihood comparisons are closely connected to the residual sum of squares (RSS). The reduction in RSS when moving to the larger model reflects increased explanatory power, and nested testing can be expressed equivalently using familiar sums-of-squares-based statistics.

2.2 Degrees of freedom and model complexity

The increased flexibility of the full model is captured by the number of additional free parameters. In classical settings, the difference in degrees of freedom between reduced and full models determines the reference degrees of freedom for certain test statistics and informs penalty terms in information criteria.

2.3 Null hypotheses induced by nesting

Nesting induces a specific null hypothesis: the additional parameters in the full model are unnecessary and equal to their constrained values implied by the reduced model. Testing therefore evaluates whether the data support relaxing those constraints.

2.4 Assumptions and regularity conditions

Many likelihood-based results depend on conditions such as:

Identifiability of model parameters (distinct parameter values correspond to distinct distributions).
Parameters not lying on the boundary of the parameter space under the null.
Smoothness and differentiability of likelihood around the optimum.
Correct model form for the response distribution (e.g., appropriate variance structure).

When these conditions fail, the reference distribution used for p-values may become inaccurate, requiring modified tests or resampling approaches.

3 Likelihood-ratio testing (LRT)

The likelihood-ratio test compares a reduced model to a full nested model using a statistic built from the likelihood ratio.

3.1 Test statistic construction

Let \(D = 2(\ell_F - \ell_R)\). This “twice the log-likelihood improvement” statistic is nonnegative and equals zero when the models fit equally well in likelihood terms.

3.1.1 Comparing constrained vs unconstrained fits

Operationally, one fits both models to the same data. After obtaining maximum likelihood estimates for each, the log-likelihoods are plugged into \(D\). The reduced model’s constraints define the null scenario; the full model relaxes them, allowing greater fit if the constraints are not supported by data.

3.2 Reference distribution under the null

Under standard regularity conditions and for large samples, \(D\) is asymptotically distributed as a chi-square random variable with degrees of freedom equal to the number of restrictions (the parameter count difference between full and reduced models). This yields a basis for computing p-values.

3.3 p-values and decision rules

Given an observed value \(D_{\text{obs}}\), the p-value is the probability (under the null) of seeing a statistic at least as large as \(D_{\text{obs}}\). Analysts often report p-values alongside effect measures, then interpret results relative to a pre-specified significance threshold, while recognizing that statistical significance does not guarantee practical usefulness.

3.4 Interpreting effect size in addition to significance

Because nested tests can be highly sensitive in large samples, it is common to supplement p-values with measures such as:

The magnitude of log-likelihood gain.
Changes in predicted probabilities or fitted values for relevant covariate ranges.
For linear Gaussian models, the increase in \(R^2\) or reduction in error variance.

These help distinguish a model that is merely “detectably better” from one that provides a meaningful improvement.

3.5 Small-sample considerations

In smaller samples, asymptotic chi-square approximations may be poor. Alternatives include:

Using finite-sample corrections or exact methods where available.
Employing parametric bootstrap to approximate the null distribution of the statistic.
Checking sensitivity by comparing results across reasonable modeling choices.

The appropriate choice depends on model type and the severity of regularity violations.

4 Information-criterion approaches

Information criteria compare models using a trade-off between goodness of fit and complexity. They are often used when the analyst wants a selection rule rather than a hypothesis test with a formal null distribution.

4.1 AIC for model selection

The Akaike Information Criterion is \( \mathrm{AIC} = -2\ell + 2k\), where \(\ell\) is the maximized log-likelihood and \(k\) is the number of estimated parameters. Lower AIC indicates a preferable balance between fit and complexity.

4.2 BIC for model selection

The Bayesian Information Criterion is \( \mathrm{BIC} = -2\ell + k\log(n)\), with \(n\) the sample size. Because the penalty grows with \(n\), BIC tends to favor simpler models more strongly than AIC as the dataset increases.

4.3 Relationship to likelihood and complexity penalties

Both AIC and BIC combine two ingredients:

A fit component (via \(-2\ell\)).
A penalty proportional to model size.

The difference between AIC and BIC lies in how severely they penalize additional parameters. Interpreting results typically involves comparing criterion values across candidate models rather than treating a single value as meaningful in isolation.

4.4 Choosing among multiple nested alternatives

When there are several nested models (e.g., polynomial degree 1 through 5), information criteria can be computed for each and the best-scoring model selected. Even though the models are nested, the criteria do not require a formal sequence of hypothesis tests; instead they provide a comparative ranking based on the penalized objective.

4.5 Practical guidelines for reporting

Reporting often includes:

The criterion used (AIC or BIC).
The difference relative to the best model (e.g., \(\Delta\)AIC or \(\Delta\)BIC).
The number of parameters and sample size.
Brief justification for the candidate set and whether the models are fit using the same data preparation and estimation approach.

Where predictive goals dominate, analysts may also supplement criterion-based selection with out-of-sample evaluation.

Nested comparison connects to other test families and to model refinement strategies beyond direct likelihood-ratio testing.

5.1 Score tests and their connection to nested models

Score tests evaluate constraints at the reduced model fit without requiring maximum likelihood estimates under the full model. They use the gradient (score) of the log-likelihood under the reduced model and can be computationally convenient or numerically stable in some settings. Under suitable conditions, the resulting reference distribution is again asymptotically chi-square with degrees of freedom corresponding to the number of restrictions.

5.2 Wald tests vs likelihood-ratio tests

Wald tests examine whether parameter estimates in the full model satisfy the constraints, using an estimate of their variance to form a standardized statistic. Compared with likelihood-ratio tests, Wald tests can be more sensitive to estimation quality when parameters are poorly estimated or constraints approach boundary values. When regularity conditions hold, different approaches often agree asymptotically.

In hierarchical model building, one may add terms in stages (e.g., blocks of covariates) and compare the model after each addition. Sequential nested testing yields an interpretable refinement path, but it can complicate control of error rates if many tests are performed. Even if overall significance is not formally controlled, sequential comparisons can still be useful for understanding which blocks contribute to fit.

5.4 Non-nested vs nested: conceptual contrasts

Nested model comparison is tailored to “subset” relationships between models. Non-nested comparison involves different parameterizations not obtainable by constraints from each other (e.g., two different distribution families). In that case, likelihood-ratio tests do not apply directly, and alternative strategies like generalized information criteria, cross-validation, or specific non-nested tests are needed.

5.5 Extensions for generalized linear models

For generalized linear models (GLMs), likelihood-based nesting and the likelihood-ratio test remain common tools. However, the interpretation of “deviance” and degrees of freedom depends on the link function, distribution, and dispersion structure. Asymptotic approximations are often used, with extra care when sample sizes are limited, separation occurs in logistic models, or dispersion parameters require special treatment.

6 Implementation workflow

A practical workflow helps ensure that comparisons are valid and reproducible.

6.1 Specifying candidate models

Start by listing the reduced model and the corresponding full model with a clear mapping of constraints. Candidate sets should be coherent: the reduced model should be derivable by fixing parameters implied by the hypothesis of interest. Model formulae should be built so that nesting is unambiguous (e.g., by adding predictors or interactions in a structured way).

6.2 Fitting both nested models consistently

Fit the reduced and full models using the same response, covariate data, and estimation settings. For likelihood-ratio testing and information criteria, both fits must be obtained via maximum likelihood (or an equivalent objective) under the respective models. In iterative algorithms, consistent initialization and convergence criteria can improve comparability.

6.3 Checking convergence and fit diagnostics

Convergence failures, warnings about nonidentifiability, or extreme parameter estimates can invalidate comparisons. Diagnostics may include:

Convergence status and iteration counts.
Examination of fitted values and residual patterns (as appropriate for the model class).
Stability under mild perturbations of starting values.

If problems arise, alternative optimizers, regularization (with careful interpretation), or different approximations may be considered.

6.4 Computing the comparison statistic or criterion

For LRT, compute \(D = 2(\ell_F-\ell_R)\) from the maximized log-likelihoods. For information criteria, compute AIC or BIC for each model and compare values. When multiple candidate models are used, store log-likelihoods and parameter counts so results can be audited.

6.5 Visualizing fit improvements where applicable

Visualization can complement numerical criteria. For regression models, plots may show predicted curves, residual distributions, or the effect of adding predictors. For likelihood-ratio outcomes, one may visualize the change in fitted values across covariate patterns. Such visuals are descriptive and should not replace formal criterion comparisons.

7 Common pitfalls and limitations

Nested comparison is powerful, but errors often stem from assumption violations, model mismatch, or misuse of results.

7.1 Overfitting despite statistical tests

A significant likelihood improvement indicates the full model fits better than the reduced model under the assumed data-generating structure, not necessarily that it predicts better out of sample. If the candidate space is large or flexible, even correct nested testing may favor over-parameterized models. Predictive checks and validation are therefore important, especially when the goal is generalization.

7.2 Violated assumptions and model misspecification

If the model family or distributional assumptions are wrong (e.g., incorrect error variance structure), the likelihood comparisons may attribute improvements to wrong mechanisms. Misspecification can also affect the validity of p-values derived from asymptotic distributions. Practical mitigation includes residual diagnostics and sensitivity analyses.

7.3 Identifiability issues and boundary parameters

When parameters are not identifiable or constrained parameters lie on the boundary under the null, standard chi-square reference results may not hold. This is common in mixture-like behavior, variance component testing, or models where certain effects can be zero with nonstandard implications. In such cases, specialized tests or bootstrap calibration are often preferable.

7.4 Heteroscedasticity and dependence concerns

For methods based on likelihood under constant variance assumptions, heteroscedasticity can distort inference. Dependence in the data (e.g., time series correlation) can also invalidate standard error assumptions that underpin theoretical results. While nested comparison can still be performed, robust alternatives or modeling the dependence structure (e.g., using appropriate correlation models) may be needed for defensible inference.

7.5 Multiple comparisons across many nested models

Comparing many nested models using repeated tests increases the chance of false discoveries if error rates are not controlled. Information criteria naturally reduce the need for a sequence of formal tests, but they still involve multiple comparisons in a broader sense. When hypothesis testing is used repeatedly, adjustments or hierarchical decision rules may be necessary to keep interpretation coherent.

8 Practical examples (non-controversial)

The following examples illustrate nested comparisons in familiar, well-behaved settings.

8.1 Comparing polynomial regression orders

A common nested setup is polynomial regression of increasing degree. For instance, degree \(d\) uses terms \(\{1,x,x^2,\dots,x^d\}\). Comparing degree 2 versus degree 3 is a nesting relationship: the reduced model is obtained by constraining the cubic coefficient to zero. An LRT evaluates whether the additional cubic term improves likelihood; AIC/BIC rank the degrees by penalized fit.

8.2 Testing added predictors in linear regression

Consider a linear regression with predictors \(X_1\) and \(X_2\). The reduced model includes \(X_1\) only, while the full model includes both \(X_1\) and \(X_2\). Nesting holds because the reduced model is recovered by setting the coefficient of \(X_2\) to zero. Under normal-error assumptions, the LRT corresponds to the classical partial F-test in this setting.

8.3 Comparing nested logistic regression specifications

In logistic regression, a nested comparison might add an interaction term. For example, compare a model with main effects \(x_1\) and \(x_2\) to a model with an added interaction \(x_1x_2\). The interaction coefficient plays the role of an additional parameter; the reduced model is recovered when that coefficient is constrained to zero. Likelihood-ratio testing and information criteria can be used, with extra attention to convergence and potential separation in small datasets.

8.4 Nested ANOVA-style comparisons

ANOVA can be interpreted as nested model comparison in balanced designs. A reduced model may omit a factor or omit an interaction, while the full model includes it. The classic F-tests arise from the comparison of sums of squares consistent with nested likelihood-based reasoning. When assumptions like normality and equal variance are reasonably met, these tests provide interpretable model comparisons.

8.5 Small illustrative simulation study

A simulation example can show how nested comparisons behave as sample size changes. One can generate data from a known regression model, fit nested alternatives that include or exclude a true additional term, then record rejection rates for LRT and the selection frequency for AIC/BIC. Such a study typically demonstrates:

Increasing sample size improves power to detect truly needed terms.
Under the null (when the extra term is absent), rejection rates stay near the nominal level only when regularity conditions hold well.
Information criteria may favor different complexity levels depending on whether AIC or BIC is used.

9 Reporting and reproducibility

Clear reporting supports accurate interpretation and makes results auditable.

9.1 What to report: statistics, degrees of freedom, p-values/criteria

For LRT-based comparisons, report:

The observed test statistic \(D\).
The degrees of freedom corresponding to the number of restrictions.
The p-value (and whether asymptotic reference was used).

For AIC/BIC, report the criterion values for each candidate model and the selected model. Including \(\Delta\)AIC or \(\Delta\)BIC helps readers understand how strongly the selection favors the winner.

9.2 Model formula documentation

Document the explicit model structure for both reduced and full models. This includes how predictors, interactions, transformations, and offsets are represented. In nested comparisons, it is helpful to state the constraints that link the models (e.g., “reduced model sets \(\beta_3=0\)”).

9.3 Reproducible code and parameter settings

Reproducibility typically requires:

The software and version.
Estimation method details (e.g., maximum likelihood settings).
Convergence criteria and any optimizer options.
Random seeds for simulations, bootstrapping, or cross-validation.

Storing model objects and the logged fit statistics (log-likelihood, parameter counts) improves the transparency of the comparison.

9.4 Interpreting results for the modeling goal

Interpretation should align with the stated purpose. If the goal is inference, emphasize whether evidence supports relaxing the constraints and how large the improvement is. If the goal is prediction, complement nested comparison with out-of-sample evaluation. Avoid equating statistical significance with practical relevance without considering effect size and predictive impact.

10 Glossary and quick reference

A short reference helps consolidate terminology and operational steps.

10.1 Nested model terminology

Reduced model: the simpler, constrained model within a nested pair.
Full model: the more flexible model obtained by relaxing the constraints.
Restrictions: the parameter equalities or fixed values defining the reduced model.
Test statistic: the computed quantity (e.g., likelihood-ratio \(D\)) used to assess evidence under nesting.

10.2 Key formulas and symbols

Likelihood-ratio statistic: \(D = 2(\ell_F-\ell_R)\).
AIC: \(\mathrm{AIC} = -2\ell + 2k\).
BIC: \(\mathrm{BIC} = -2\ell + k\log(n)\).

Here \(\ell_F,\ell_R\) are maximized log-likelihoods; \(k\) is the number of estimated parameters; \(n\) is sample size.

10.3 Rule-of-thumb checklist for use

Confirm the reduced model is obtainable by explicit constraints from the full model.
Fit both models consistently and verify convergence.
Use LRT when asymptotic conditions appear reasonable; otherwise consider bootstrap calibration.
Use AIC/BIC for model selection and compare multiple candidates systematically.
Report the comparison quantities (statistics, degrees of freedom, p-values, or criterion values) and interpret in light of effect size and the intended goal (inference vs prediction).