1 Introduction to Information-Theoretic Model Selection

1.1 The problem of comparing statistical models

Model selection addresses the question of which statistical model best describes observed data among a set of candidate alternatives. In practice, models may differ in functional form, included predictors, distributional assumptions, or latent structure. Because a more flexible model typically improves fit, selection criteria must incorporate both adequacy of fit and the risks introduced by extra complexity.

1.2 Balance between fit and complexity

Information-theoretic criteria formalize the trade-off between how well a model explains data and how complicated it is. Complexity is commonly operationalized through the effective number of parameters, which discourages overfitting—selecting a model that tracks noise rather than signal. The resulting criteria typically combine a goodness-of-fit term derived from likelihood with a penalty term that increases with complexity.

1.3 When “lower is better” applies

Both AIC and BIC are typically reported so that smaller values indicate a preferable model, given that the criteria are computed consistently across the candidate set. This “lower is better” convention holds for the standard forms of AIC and BIC used in likelihood-based settings, where the criterion value is computed from the maximized log-likelihood plus a positive penalty.

2 Likelihood Foundations

2.1 Likelihood and log-likelihood in model comparison

Let a model with parameters \(\theta\) imply a likelihood \(L(\theta)=p(y\mid \theta)\) for data \(y\). Model selection criteria rely on the maximized likelihood (or equivalently maximized log-likelihood) over parameters. Using log-likelihood is convenient because it converts products of probabilities into sums and aligns naturally with information measures used in theoretical derivations.

2.2 Deviance, entropy intuition, and goodness-of-fit

Goodness-of-fit can be expressed through quantities related to deviance, often defined as a function of \(-2\) times the log-likelihood. Under common regularity assumptions, models with larger maximized log-likelihood (or smaller deviance) fit the data better. In an information-theoretic interpretation, the log-likelihood connects to notions such as cross-entropy: models that assign higher probability to observed outcomes tend to have lower expected information loss.

2.3 Regularity conditions and practical caveats

The theoretical justifications for AIC and BIC typically assume conditions such as smoothness of the likelihood, identifiability, and an adequately large sample size. In real applications, violations can occur—for instance, when parameters lie near boundaries, the model is weakly identified, or the likelihood surface is irregular. These issues can distort the intended meaning of the penalty terms, making results less reliable if used mechanically.

3 AIC: Akaike Information Criterion

3.1 Definition of AIC

Akaike Information Criterion is defined (in its common likelihood-based form) as \[ \mathrm{AIC} = -2\log L(\hat{\theta}) + 2k, \] where \(L(\hat{\theta})\) is the likelihood evaluated at the maximum likelihood estimate \(\hat{\theta}\), and \(k\) denotes the number of estimated parameters (or an effective parameter count in some extensions).

3.2 Derivation intuition (expected predictive performance)

AIC is often motivated by the idea of minimizing expected discrepancy between the fitted model and the unknown data-generating process. While the underlying data-generating mechanism is not directly observable, AIC can be interpreted as estimating the expected out-of-sample predictive error under certain assumptions. This perspective favors models that generalize well rather than merely achieving the best in-sample fit.

3.3 Penalty term and effective number of parameters

The penalty \(2k\) increases linearly with the parameter count, reflecting that each additional parameter consumes degrees of freedom and can capture spurious patterns. In models where parameters are not equally influential—such as cases with redundancies, constraints, or nonstandard parameterizations—the relevant notion of “effective” complexity may differ from the raw parameter tally. This is why practical implementations sometimes use generalized definitions of \(k\).

3.4 Corrected AIC variants (e.g., AICc)

When samples are small relative to model complexity, the standard AIC approximation can be biased. Corrected variants such as AICc introduce an adjustment that increases the penalty for complexity, improving performance in finite-sample regimes. AICc is widely used when \(n/k\) is not large, where \(n\) is the sample size.

3.5 Small-sample considerations

Small samples amplify the impact of assumptions behind AIC’s theoretical approximation. Consequences include unstable selection, especially when candidate models are close in fit or when maximum likelihood estimates have high variance. In such situations, analysts typically pair AIC-based selection with diagnostic checks, resampling-based stability assessment, and, when appropriate, use of AICc.

4 BIC: Bayesian Information Criterion

4.1 Definition of BIC

Bayesian Information Criterion is typically defined as \[ \mathrm{BIC} = -2\log L(\hat{\theta}) + k\log n, \] where \(n\) is the sample size. As with AIC, the criterion is computed from the maximized likelihood, but the complexity penalty grows with \(\log n\) rather than linearly with \(k\).

4.2 Derivation intuition (approximate marginal likelihood)

BIC is motivated by approximating a Bayesian model comparison quantity: the marginal likelihood (also called model evidence), integrated over parameter values with a prior. Under regularity conditions and large-sample asymptotics, the log marginal likelihood can be approximated by a form equivalent to \(-\tfrac12\)BIC up to additive constants that do not depend on the candidate model. This yields BIC as a large-sample approximation to Bayesian evidence.

4.3 Penalty term scaling with sample size

The term \(k\log n\) penalizes complexity more strongly as the sample size increases. This property makes BIC more conservative: it tends to select simpler models when the true predictive signal is not strong enough to justify extra parameters. In large datasets, the growing penalty can dominate differences in fit, steering selection toward parsimony.

4.4 Relationship to Bayesian model comparison

Although BIC is not itself a full Bayesian calculation, it is closely related to Bayesian comparison by design. In Bayesian frameworks, evidence naturally incorporates a trade-off between fit and complexity through integration over parameters. BIC provides a computationally efficient shortcut under assumptions that justify its asymptotic approximation.

5 Comparison Between AIC and BIC

5.1 Different penalty strengths and typical behaviors

AIC and BIC usually disagree because their penalties scale differently. AIC uses a constant penalty \(2k\), while BIC uses \(k\log n\), which is larger than \(2k\) for sufficiently large \(n\). As a result, AIC more readily chooses models with additional parameters, often aligning with predictive emphasis, whereas BIC more readily selects smaller models, often aligning with consistency in certain asymptotic regimes.

5.2 Practical scenarios where they may disagree

Disagreement is most common when candidate models fit similarly in-sample but differ in complexity. With moderate sample sizes, whether the incremental improvement in likelihood outweighs the penalty determines the winner. BIC’s heavier penalty tends to suppress marginally beneficial parameters, while AIC may retain them if they contribute to higher likelihood even modestly.

5.3 Interpretation of results and stability checks

Neither criterion is universally “correct”; each expresses a particular principle and set of assumptions. A practical approach is to inspect how much the criteria differ across models, whether the selected model remains stable under resampling, and whether predictive diagnostics support the choice. When two models have close criterion values, uncertainty about selection is often warranted rather than overconfident interpretation.

6 Computing AIC/BIC in Common Models

6.1 Linear regression and generalized linear models (conceptual)

In linear regression with Gaussian errors, the maximized likelihood is determined by the residual sum of squares and estimated variance. AIC and BIC can then be computed from \(-2\log L(\hat{\theta})\) plus the relevant parameter count. In generalized linear models, the likelihood depends on the mean-variance relationship and the chosen link function; once maximum likelihood estimates are obtained, the same general criterion formula applies using the maximized log-likelihood.

6.2 Maximum likelihood estimation workflow

Computing AIC/BIC typically follows a workflow: (1) specify the candidate model, (2) fit the model by maximum likelihood to obtain \(\hat{\theta}\), (3) extract the maximized log-likelihood, and (4) compute the penalty using the parameter count \(k\) and sample size \(n\). Consistency matters: the same likelihood basis (including how dispersion parameters are treated) and consistent definition of \(k\) must be used across candidates.

6.3 Handling constrained or nested models

Candidate models may be nested (one model is a special case of another) or may involve constraints. For nested models, maximum likelihood fitting under each model yields different estimates and different log-likelihood values, enabling direct criterion computation. For constraints, parameter counting can require care: some implementations treat constrained parameters differently or use effective degrees of freedom. When constraints are strong, the nominal parameter count may not reflect the true flexibility relevant to the criterion.

6.4 Mixture and latent-variable models (practical considerations)

Mixture models and latent-variable models often have likelihood surfaces with multiple local maxima and can exhibit identifiability difficulties. In such settings, the computed log-likelihood can depend on initialization and optimization quality. Since AIC and BIC rely on the maximized log-likelihood, practitioners usually run multiple fits with different starting values and compare the best achieved likelihood across models, while also considering whether the fitted solutions are stable.

7 Assumptions and Limitations

7.1 Model class and likelihood specification requirements

AIC and BIC require a likelihood specified up to estimated parameters. If the likelihood is misspecified—for example, an incorrect error distribution or an omitted dependence structure—the log-likelihood no longer measures fit in the intended way. Both criteria then become selectors for the “best among wrong models” rather than reliable indicators of truth.

7.2 Non-nested model comparisons

AIC and BIC are frequently used to compare non-nested models, where one cannot be obtained from the other by removing terms or relaxing parameters. The criteria can still be computed because they depend only on maximized likelihood and parameter counts. However, the interpretation must remain cautious: comparing models with qualitatively different assumptions can yield values that reflect relative fit under those assumptions rather than a universal notion of correctness.

7.3 Overdispersion and mis-specified likelihoods

In count data or other settings where variance may exceed what the assumed model predicts, overdispersion can reduce the credibility of the likelihood-based fit. Since both criteria use the likelihood magnitude, miscalibration of variance can lead to systematic preference for certain model structures. A common remedy is to use a more appropriate likelihood (e.g., negative binomial) or add dispersion parameters, but the suitability must be assessed for each application.

7.4 Correlated data and dependence structures

If observations are correlated (for example, time series dependence, clustering, or repeated measures) and the model fails to account for it, the effective information content differs from what the criterion assumes. The penalty terms based on nominal parameter counts may no longer properly reflect complexity relative to dependence. Extensions such as likelihoods that include correlation structures, appropriate random effects, or dependence terms are typically needed for meaningful comparisons.

8 Workflow for Using AIC/BIC

8.1 Selecting the candidate model set

Begin by enumerating plausible models grounded in theory, prior empirical patterns, or experimental design constraints. The candidate set should be limited enough to avoid enormous search spaces but broad enough to capture major alternatives. It is often beneficial to include both simple baselines and richer variants so that selection has meaningful competition.

8.2 Estimation, diagnostics, and convergence

Fit each candidate model using maximum likelihood methods compatible with the assumed likelihood. Diagnostics help verify that optimization converged to a suitable solution and that residual behavior or diagnostic measures do not show glaring failures. In models with latent structure or potential nonconvexity, multiple starting points and convergence checks are especially important.

8.3 Ranking models and reporting differences

Compute AIC and/or BIC for each model and rank by lowest criterion value. Reporting should include not only the top choice but also the relative differences between leading models, since small gaps can indicate ambiguity. Presenting results in a transparent table with the log-likelihood and penalty components can help readers interpret why a model is preferred.

8.4 Sensitivity analysis (resampling and refitting)

Because model selection can be sensitive to sample fluctuations, resampling-based checks such as bootstrap or cross-validation (used as diagnostics, even if the selection criterion is AIC/BIC) can assess robustness. Another approach is to refit under alternative optimization settings or slight perturbations in preprocessing to determine whether the preferred model changes. Stable preferences across such checks strengthen confidence.

Beyond AIC and BIC, there exist other information criteria that alter the penalty term or derive from different optimality targets. Examples include variants designed to improve performance under small samples or to align more closely with Bayesian evidence approximations. These criteria can be useful when the assumptions behind AIC/BIC appear mismatched to the data regime.

9.2 Cross-validation vs AIC/BIC (conceptual comparison)

Cross-validation directly estimates predictive performance by partitioning data into training and validation subsets. AIC and BIC instead use a likelihood-based approximation to expected predictive error (AIC) or evidence approximation (BIC). Cross-validation can be more computationally intensive but is often more flexible regarding model misspecification and dependence structures. In practice, cross-validation may serve as a complementary check on AIC/BIC-selected models.

9.3 Model averaging and evidence-weighted approaches

When several models have comparable criterion values, selecting only one model may waste information. Model averaging combines predictions from multiple candidates using weights derived from a criterion-related measure, sometimes based on differences in AIC or BIC. Evidence-weighted approaches aim to reflect relative support across models, which can improve stability and reduce sensitivity to a single modeling choice, particularly in borderline cases.