1 Definition and intuition

Information criteria are statistical measures designed to compare candidate models using a score that balances (i) how well the model fits the observed data and (ii) how much complexity the model introduces. The resulting ranking is intended to approximate which model would yield better predictive performance on new, unseen data, rather than simply maximizing agreement with the particular sample.

1.1 Likelihood-based model scoring

In likelihood-based settings, an information criterion typically starts from the maximized log-likelihood of a fitted model. Higher likelihood indicates that the model can explain the data more plausibly, while a complexity term adjusts the score downward when the model has more parameters or greater flexibility. This construction allows models to be compared on a common numerical scale.

1.2 Fit versus complexity trade-off

The core idea is to prevent the selection process from favoring models that improve fit only because they are flexible enough to adapt to random noise. The complexity penalty grows with model size, so the chosen model is the one that achieves the best net compromise between explanatory power and parsimony.

1.3 Regularity assumptions and interpretation

Most information criteria are derived under standard regularity conditions for likelihood-based estimation (such as smoothness of the likelihood and identifiability of parameters). Under these assumptions, the criterion’s asymptotic behavior can be interpreted in terms of an expected discrepancy between the fitted model and the unknown data-generating process. When assumptions are strained, the criteria can still be used as heuristics, but their theoretical meaning becomes less reliable.

2 Core information criteria

Several widely used information criteria share the same general structure—an improvement in fit can be offset by a penalty for complexity—but differ in how they choose the strength and form of the penalty.

2.1 Akaike Information Criterion (AIC)

AIC is among the most common tools for model selection in likelihood-based models. It is especially associated with aiming at predictive accuracy.

2.1.1 Derivation from Kullback–Leibler divergence

AIC can be motivated by considering the Kullback–Leibler (KL) divergence between the true data-generating distribution and a candidate model’s implied distribution. Under likelihood-based estimation and regularity conditions, AIC provides an approximately unbiased estimate of the expected KL discrepancy (up to an additive constant). Consequently, the model with the smallest AIC is the one with the best estimated predictive discrepancy under this framework.

2.1.2 Practical computation and common variants (AICc)

For many models, AIC is computed as \[ \text{AIC} = -2\ell(\hat\theta) + 2k, \] where \(\ell(\hat\theta)\) is the maximized log-likelihood and \(k\) is the number of estimated parameters. When sample sizes are small relative to model size, AIC is adjusted to reduce small-sample bias. This yields the corrected version AICc, which adds an extra term that increases the effective penalty for larger models. AICc is commonly recommended when \(n/k\) is not large.

2.2 Bayesian Information Criterion (BIC)

BIC is another prominent criterion and often appears in contexts where asymptotic model evidence arguments are invoked.

2.2.1 Connection to model evidence and asymptotics

BIC can be derived from approximations to the log marginal likelihood (model evidence) under Bayesian principles. For regular models and large samples, the leading term of the approximation implies a criterion of the form \[ \text{BIC} = -2\ell(\hat\theta) + k\log(n), \] where \(n\) is the sample size. The \(\log(n)\) growth makes BIC penalize complexity more strongly as data accumulate.

2.2.2 Interpretation of the penalty term

Because BIC’s penalty increases with \(\log(n)\), it tends to favor simpler models when \(n\) is large, sometimes leading to selection of the true model (when certain conditions hold). In contrast, weaker penalties can maintain the inclusion of additional parameters for longer, potentially improving predictive performance at the cost of selecting more complex models.

Beyond AIC and BIC, several alternatives adjust the penalty structure to target different theoretical objectives or accommodate different asymptotic regimes.

2.3.1 Takeuchi Information Criterion (TIC/TIC-type)

Takeuchi’s information criterion extends AIC to scenarios where the model is potentially misspecified, so the usual assumptions used in the AIC derivation may not hold. In such settings, the penalty depends not only on parameter count but also on how the likelihood curvature and parameter sensitivity differ under the fitted model versus the data-generating process.

2.3.2 Hannan–Quinn criterion

The Hannan–Quinn (HQ) criterion provides another middle-ground penalty, typically using a penalty that grows like \(2k\log(\log(n))\) (up to constant factors). It is designed to balance between AIC-like and BIC-like behavior across sample sizes, reflecting a compromise between predictive focus and consistency properties.

3 Choosing among criteria

No single information criterion dominates in all conditions. Selection among criteria can be guided by sample size, desired behavior, and characteristics of the model family.

3.1 Guidance by sample size

When sample sizes are small, corrected criteria such as AICc often reduce bias relative to their uncorrected versions. With large \(n\), differences among criteria increasingly arise from how strongly each penalizes additional parameters, which can substantially affect the selected model.

3.2 Impact of penalty strength

The penalty term governs model selection behavior. AIC-type penalties are relatively modest, which can encourage inclusion of extra terms that improve in-sample fit in ways that may generalize. BIC-type penalties are more aggressive and can remove parameters unless the gain in likelihood is substantial. HQ and TIC-type criteria adjust penalty behavior to target different objectives, such as robustness to misspecification.

3.3 When each criterion tends to perform well

In many practical predictive modeling tasks, AIC and AICc often perform competitively because they are linked to expected predictive discrepancy. BIC can be beneficial when model parsimony is strongly valued or when large-sample behavior is expected to align with consistency goals. TIC-type approaches are typically used when concern exists that the modeling assumptions may not fully hold, and an adjustment for misspecification is desirable.

3.4 Model selection consistency considerations

Consistency refers to selecting the correct model as \(n\to\infty\) under appropriate conditions. BIC-like penalties that grow faster than AIC’s can support consistency in standard nested settings, while AIC is often associated with minimizing predictive loss rather than guaranteeing consistent identification of the “true” model. HQ offers a tunable balance, with its asymptotic penalty growth positioned between the extremes.

4 Extensions and modern adaptations

Modern applications broaden the basic framework to handle small samples, alternative likelihood forms, and more complex modeling structures.

4.1 Corrected and small-sample adjustments

Small-sample corrections modify the penalty to compensate for bias introduced by using asymptotic approximations. AICc is the best-known example; additional corrections sometimes appear in specialized variants, particularly when parameters increase quickly relative to sample size.

4.2 Conditional and quasi-likelihood settings

When the full likelihood is unavailable or difficult to specify, quasi-likelihood or conditional likelihood ideas may replace the original likelihood. Criteria can then be adapted using an effective measure of fit and a corresponding effective complexity term derived from the estimating equations or working models. These adaptations aim to preserve a similar balancing principle even when the statistical model is expressed indirectly.

Many criteria can be interpreted as special cases of penalized likelihood approaches. Under certain approximations, the information criterion score resembles a log-likelihood plus an explicit penalty proportional to model size. This connection helps contextualize why they resemble regularization choices, though information criteria are usually used for comparison rather than as optimization objectives to produce the final estimator.

4.4 Criteria under misspecification

If the model class cannot represent the true data-generating process, the theoretical justification of classic criteria can fail. Misspecification-robust versions (such as TIC-type criteria) incorporate additional sensitivity information that better reflects how the fitted model behaves relative to the unknown truth. These approaches can yield more reliable comparisons when the modeling assumptions are only approximately satisfied.

5 Applications in common modeling tasks

Information criteria are used across many modeling domains because they are computationally convenient once the fitted likelihood for each candidate is available.

5.1 Linear and generalized linear models

For linear regression with Gaussian errors, AIC and BIC reduce to familiar expressions involving the residual sum of squares and parameter count. In generalized linear models, the maximized likelihood is computed under the assumed distribution and link function, after which the information criteria apply with the chosen notion of parameter count.

5.2 Multilevel/hierarchical models (overview-level usage)

In hierarchical models, the “effective complexity” can be less obvious than a simple parameter count, since random effects and variance components contribute to flexibility. Information criteria are sometimes applied using approximate marginal likelihoods, Laplace approximations, or effective degrees of freedom. In practice, users often rely on specialized implementations that approximate how complexity should be counted for hierarchical structures.

5.3 Time series and state-space models

For autoregressive and related models, information criteria provide a fast way to compare candidate orders by fitting each order and scoring it by likelihood and penalty. In state-space models, exact likelihood evaluation may be expensive, so approximate likelihood methods (e.g., filtering-based likelihoods) can be used to compute the criteria. When likelihood approximations are used, the uncertainty in the score can affect the reliability of the ranking.

5.4 Feature selection and variable subset comparisons

When comparing models defined by different variable subsets, information criteria offer a computationally simpler alternative to exhaustive cross-validation in some settings. The typical workflow is to fit each subset model, compute the criterion, and select the lowest-scoring model. This can be helpful for screening, though it is important to recognize that model selection itself can introduce optimistic bias if the same data guide both selection and evaluation.

6 Relation to other model selection methods

Information criteria are one part of a broader toolbox that includes resampling-based approaches, hypothesis-testing frameworks, and Bayesian selection methods.

6.1 Information criteria vs. cross-validation

Cross-validation estimates predictive performance by repeatedly training on part of the data and evaluating on held-out segments. Information criteria often provide a cheaper proxy for predictive performance by using fitted likelihood and an analytic complexity penalty. Cross-validation can be more robust in situations where the criterion’s theoretical approximations are questionable, but it can be computationally intensive, especially for many candidate models.

6.2 Information criteria vs. likelihood ratio testing

Likelihood ratio tests compare nested models using a test statistic derived from likelihood differences. Information criteria also use likelihood differences but translate them into a penalized score rather than a hypothesis test with a chosen significance level. This means information criteria can be applied more naturally across non-nested models, whereas standard likelihood ratio testing is typically limited to nested structures.

6.3 Information criteria vs. Bayesian model selection (high level)

Bayesian model selection uses posterior model probabilities, often built from marginal likelihoods. Information criteria, particularly BIC, can be interpreted as large-sample approximations to log marginal likelihood (up to constants). As a result, BIC can be viewed as a computationally lighter alternative to fully Bayesian evidence computations, while still reflecting Bayesian-like penalties through asymptotic arguments.

7 Practical implementation

Successful use of information criteria depends on careful computation of likelihood values, an appropriate definition of model complexity, and attention to numerical and modeling details.

7.1 Computing from fitted likelihoods

The essential computation requires the maximized log-likelihood under each candidate model. Users must ensure that the likelihood is computed consistently across models—for example, using the same data preprocessing and the same treatment of observation structure (weights, offsets, censoring, or missingness handled in the same way). Any mismatch can distort comparisons.

7.2 Handling nuisance parameters and degrees of freedom

Nuisance parameters and variance components affect the maximized likelihood and thus influence the score indirectly. The explicit complexity term depends on how the criterion defines \(k\). In many simple models, \(k\) is the number of free estimated parameters. In more complex models (e.g., hierarchical or constrained parameterizations), a naive count may not reflect effective complexity, motivating specialized implementations or adjusted degrees-of-freedom calculations.

7.3 Numerical stability and software considerations

Likelihood computations can be unstable when probabilities are very small, parameter estimates are near boundaries, or models have separation-like issues in logistic contexts. Software often uses log-scale computations and robust optimization routines. For fair model comparison, users should confirm convergence diagnostics, identical starting procedures when appropriate, and adequate optimization tolerances across candidate models.

7.4 Diagnosing when criteria may mislead

Information criteria rely on assumptions such as approximate model regularity and meaningful likelihood evaluation. They may mislead when candidate models are nearly equivalent but poorly estimated, when sample sizes are too small for asymptotic approximations, when likelihood maximization is unstable, or when key modeling decisions differ across candidates (e.g., different transformations or missing-data strategies). In such cases, supplementary diagnostics and alternative selection methods can improve confidence.

8 Limitations and pitfalls

Despite their convenience, information criteria can fail to deliver the intended balance between fit and generalization.

8.1 Non-nested model comparisons

Information criteria can compare non-nested models, but the theoretical interpretations often assume certain regularity and effective complexity definitions that may not translate cleanly when model classes differ in qualitative ways. Differences in how models represent dependence or variance structures can make penalty terms less comparable.

8.2 Sensitivity to scaling and model formulation

Although likelihood-based scoring is invariant under many reparameterizations, practical formulations can change the effective meaning of parameters and penalties. For example, using different parametrizations or constraints may alter how \(k\) should be counted. Additionally, when models include transformed responses or different baseline specifications, likelihood values may not be directly comparable unless the data likelihoods are defined consistently.

8.3 Overfitting under weak penalties

If the penalty is too light relative to the model class’s flexibility, the criterion may select overly complex models that capture noise. AIC-like penalties can behave this way in certain high-dimensional regimes or when the candidate set is large and includes many near-redundant terms.

8.4 Assumption violations and robust alternatives

Violations of regularity conditions, strong misspecification, or dependence structures not accounted for by the fitting method can undermine criterion validity. When this is a concern, robust alternatives include using misspecification-aware criteria (TIC-type), combining with resampling-based evaluation, or relying on methods that target predictive performance more directly, such as cross-validation.

9 Worked example templates

The following templates illustrate typical workflows for comparing models with information criteria. The steps are schematic and can be adapted to different likelihood-based contexts.

9.1 Comparing candidate regression models

9.1.1 Using AIC/AICc step-by-step

  1. Specify candidate models: Choose a set of regression formulas (e.g., different sets of predictors or interaction terms) that you consider plausible.
  2. Fit each model: Estimate parameters for each candidate using the same likelihood framework (and the same data preprocessing).
  3. Compute the maximized log-likelihood: Extract \(\ell(\hat\theta)\) from each fitted model.
  4. Determine parameter count: Set \(k\) as the number of estimated free parameters in each model (including intercepts as appropriate for your model definition).
  5. Calculate the criterion:
  • If using AIC: \(\text{AIC} = -2\ell(\hat\theta) + 2k\).
  • If using AICc: apply the small-sample correction according to the standard AICc formula for your setup.
  1. Rank candidates: Select the model with the smallest criterion value. Optionally compute differences (e.g., \(\Delta\) values) to gauge how decisive the ranking is.
  2. Validate cautiously: Even if the criterion selects a model, check residual diagnostics or predictive checks to ensure the chosen model is not pathologically misfitting.

9.2 Comparing candidate time-series orders

  1. Choose an order grid: For example, consider several autoregressive orders or state-space model configurations.
  2. Fit each order: Estimate parameters for each candidate order using the same estimation method.
  3. Compute log-likelihood: Use the fitted likelihood (or an approximation consistently computed across candidates).
  4. Apply AIC/BIC/HQ: Decide which criterion suits your objective (predictive emphasis vs. stronger parsimony).
  5. Select and refit: Choose the best-scoring order and refit it, if needed, using the chosen settings.
  6. Assess adequacy: Verify residual autocorrelation or other time-series diagnostics to confirm that the selected order is not systematically underfitting.

9.3 Interpreting results and reporting choices

  • Report the criterion used: State whether AIC, AICc, BIC, or another variant was used.
  • Report model ranking: Provide the selected model and at least a brief summary of other competitive candidates if relevant.
  • Explain parameter counting: Mention how \(k\) was determined, especially for models with more complex structure.
  • Include context for decision-making: If selection is close between models, acknowledge uncertainty and consider supplementary predictive evaluation.

10 Summary and best-practice checklist

Information criteria provide efficient, likelihood-based comparisons by trading fit against complexity. Their effectiveness depends on consistent computation, appropriate penalty choice, and awareness of assumption limits.

10.1 Selecting a criterion with rationale

Choose the criterion that aligns with the intended goal and expected conditions. Use small-sample corrections when appropriate, consider misspecification-robust options if assumptions are doubtful, and interpret differences in scores in the context of sample size and model family behavior.

10.2 Reporting transparency and reproducibility

Report: (i) the exact criterion formula or software option, (ii) how model complexity \(k\) was defined, (iii) the candidate model set, and (iv) the estimation and likelihood settings. Include convergence and any approximation details that could affect likelihood values.

10.3 Common reporting standards for selected models

When presenting a selected model, provide enough detail for others to replicate the model search and scoring process. Include the selected specification, relevant parameter interpretation at a high level, and note that information criteria are used for model comparison rather than as definitive proof of correctness.