1 Definition and Mathematical Formulation

Marginal likelihood, often called the Bayesian evidence, is the probability of observed data after averaging over uncertainty about the model parameters. In Bayesian analysis, it converts parameter uncertainty into a single scalar quantity that can be used to compare models.

Consider data \(y\) (or a dataset \(y_{1:n}\)) and a model \(M\) with parameters \(\theta\). With prior density \(p(\theta\mid M)\) and likelihood \(p(y\mid \theta, M)\), the marginal likelihood is \[ p(y\mid M)=\int p(y\mid \theta, M)\,p(\theta\mid M)\,d\theta. \] This integral (or corresponding sum) averages the likelihood over the prior distribution, so models are rewarded for fitting the data but penalized when substantial prior mass corresponds to poor fits.

1.1 Bayesian model notation

A standard Bayesian model comparison setup uses:

  • Model index \(M\) (e.g., different functional forms, different likelihood families, different numbers of components).
  • Parameters \(\theta\) associated with \(M\).
  • Likelihood \(p(y\mid \theta, M)\).
  • Prior \(p(\theta\mid M)\).
  • Posterior \(p(\theta\mid y, M)\), obtained via Bayes’ rule.

The key identity linking these quantities is Bayes’ rule: \[ p(\theta\mid y, M)=\frac{p(y\mid \theta, M)p(\theta\mid M)}{p(y\mid M)}. \] The denominator is the marginal likelihood, ensuring the posterior integrates to one.

1.2 Computing marginal likelihood

The computational task is to evaluate an integral (continuous case) or a sum (discrete case) of the likelihood weighted by the prior.

1.2.1 Continuous parameter models

For continuous \(\theta\in\mathbb{R}^d\), \[ p(y\mid M)=\int_{\mathbb{R}^d} p(y\mid \theta, M)\,p(\theta\mid M)\,d\theta. \] Often, closed forms exist only for special choices of likelihood and prior (conjugate families). Otherwise, numerical approximation is required.

1.2.2 Discrete parameter models

If parameters take values in a countable set, \(\theta\in\{\theta_1,\theta_2,\dots\}\), \[ p(y\mid M)=\sum_{\theta} p(y\mid \theta, M)\,p(\theta\mid M). \] Discrete sums also arise when marginalizing latent variables or model indicators with a finite number of states.

1.3 Relationship to likelihood and posterior

Marginal likelihood is closely related to both likelihood and posterior but differs from each in an important way:

  • The likelihood \(p(y\mid \theta, M)\) conditions on a fixed parameter value.
  • The posterior \(p(\theta\mid y, M)\) updates the prior using the likelihood, but does not itself directly compare models.
  • The marginal likelihood \(p(y\mid M)\) averages the likelihood over the prior, and therefore reflects how plausibly the data can be generated under the entire prior-supported parameter range.

Using Bayes’ rule, the evidence can be interpreted as a normalization constant that calibrates the joint model \(p(y,\theta\mid M)\) into a posterior distribution.

2 Marginal Likelihood in Bayesian Model Comparison

Marginal likelihood enables Bayesian model selection by quantifying how well each candidate model explains the data given its prior assumptions. When comparing two models, the evidence ratio is central.

2.1 Bayes factors

For models \(M_1\) and \(M_2\), the Bayes factor is \[ \mathrm{BF}_{12}=\frac{p(y\mid M_1)}{p(y\mid M_2)}. \] A Bayes factor larger than one indicates that, relative to prior odds, model \(M_1\) has stronger support from the data. Because both numerator and denominator integrate over parameter uncertainty, Bayes factors automatically incorporate a form of complexity penalty.

2.2 Evidence interpretation and model ranking

Evidence values are not universally comparable across datasets of different sizes, but within a fixed dataset and set of models, higher \(p(y\mid M)\) (or higher log-evidence) corresponds to stronger support. Practitioners typically compare log-evidence to improve numerical stability: \[ \log p(y\mid M). \] Model ranking is then based on relative evidence or the induced Bayes factors.

2.3 Priors’ influence on evidence

Because marginal likelihood integrates the likelihood against the prior, it is sensitive to how priors allocate probability mass. Two models with identical likelihood families but different prior scales can yield different evidence values. This sensitivity is often viewed as desirable: it reflects the model’s stated uncertainty. However, it also means that prior choices should be justified and, when possible, assessed via sensitivity analysis.

In many applications, hyperparameters that control prior tightness can materially shift the marginal likelihood, sometimes dominating subtle differences in fit. Care is therefore needed when priors are weakly specified or chosen purely for convenience.

2.4 Occam’s razor in evidence-based selection

Occam’s razor emerges from the averaging structure of the evidence. A model with parameters that can match the data only in a small region of parameter space tends to have lower evidence than a model that achieves comparable fit over a broader region. This occurs because the integral effectively rewards models whose likelihood is high over substantial prior mass, while models requiring precise parameter tuning are less favored.

In this sense, marginal likelihood balances goodness-of-fit with “effective complexity,” without requiring an explicit penalty term like those used in frequentist criteria.

Marginal likelihood is closely connected to predictive performance because it is the normalizing factor in the posterior and appears in predictive densities.

3.1 Posterior predictive distribution

The posterior predictive distribution for a future observation \(y^{\ast}\) under model \(M\) is \[ p(y^{\ast}\mid y, M)=\int p(y^{\ast}\mid \theta, M)\,p(\theta\mid y, M)\,d\theta. \] While this expression does not directly equal marginal likelihood, both quantities rely on the same model components (likelihood and priors). In fact, marginal likelihood determines how the posterior itself is weighted relative to other models.

3.2 Cross-validation perspectives

Cross-validation assesses predictive accuracy by training on a subset of data and evaluating on held-out observations. Although cross-validation does not compute evidence in closed form, it is conceptually aligned with Bayesian predictive behavior:

  • Evidence-based comparisons use the whole dataset to compute \(p(y\mid M)\).
  • Cross-validation approximates how well the model predicts unseen data.

In some settings, the log-evidence and predictive scores are related, particularly when models are compared using approximations to predictive likelihood.

3.3 Information-theoretic interpretations

Evidence can be viewed through an information-theoretic lens as summarizing how well a model’s prior predictive distribution matches the observed data. Since the marginal likelihood is the probability the model assigns to the realized data, comparing evidences corresponds to comparing which model’s prior predictive distribution better explains what actually occurred.

This viewpoint connects marginal likelihood to notions like expected log score and divergence between distributions, though the exact mapping depends on whether one considers prior predictive performance or posterior-updated interpretations.

4 Estimation Methods for Marginal Likelihood

In practice, direct computation of marginal likelihood is rarely feasible for complex models. A range of estimation strategies exist, trading bias, variance, and computational cost.

4.1 Analytical conjugate cases

For conjugate priors, the posterior and evidence can be derived in closed form. Classic examples include:

  • Normal likelihood with normal prior (conjugate mean estimation).
  • Binomial likelihood with beta prior.
  • Exponential family models with matching conjugate priors.

In conjugate settings, the evidence integral collapses into ratios of normalizing constants, producing exact marginal likelihood values and stable numerical results.

4.2 Numerical integration and quadrature

When dimensionality is moderate and the integrand is well-behaved, numerical quadrature methods approximate the evidence by evaluating the integrand at selected points and weighting them appropriately. Common approaches include grid-based integration and adaptive quadrature.

However, the computational burden increases rapidly with dimension (the “curse of dimensionality”), making quadrature impractical for many Bayesian hierarchical or latent-variable models.

4.3 Monte Carlo estimators

Monte Carlo methods approximate integrals by sampling. Let \(\theta^{(s)}\) be drawn from an appropriate distribution related to the prior or posterior. Estimators then average likelihood terms to approximate \(p(y\mid M)\). Efficiency depends on how well the proposal distribution covers regions where the likelihood is large.

4.3.1 Importance sampling

A common identity expresses evidence as an expectation: \[ p(y\mid M)=\mathbb{E}_{q(\theta)}\left[\frac{p(y\mid \theta, M)\,p(\theta\mid M)}{q(\theta)}\right], \] where \(q(\theta)\) is a proposal distribution that is easy to sample from. When \(q(\theta)\) approximates the posterior well, the estimator can be effective. Poorly chosen proposals can lead to high variance and unreliable estimates.

4.3.2 Bridge sampling

Bridge sampling constructs an estimator that “bridges” between the posterior and prior (or between two distributions) to reduce variance. The method typically leverages both prior and posterior samples, aiming to stabilize the contribution of regions that dominate the integral. Bridge sampling is widely used because it often outperforms naive importance sampling, especially in moderate-dimensional problems.

4.3.3 Annealed importance sampling

Annealed importance sampling introduces a sequence of intermediate distributions between prior and posterior. Rather than moving directly from one distribution to another, the method gradually “anneals” using a temperature schedule, producing incremental updates that can improve overlap between successive distributions. This often increases computational cost but can yield more stable evidence estimates for challenging posteriors.

4.4 Nested sampling

Nested sampling estimates the evidence by iteratively replacing the lowest-likelihood samples with new draws constrained to higher likelihood levels. The algorithm produces an estimate of the evidence by accumulating contributions over increasing likelihood thresholds. This approach can be effective for multimodal or sharply peaked posteriors, though performance depends on tuning and the model’s geometry.

4.5 Variational approximations for evidence

Variational inference approximates the posterior with a tractable family \(q(\theta)\) and optimizes an objective related to the evidence. While standard variational approaches yield a lower bound on \(\log p(y\mid M)\) (often called the evidence lower bound), additional steps may be used to improve approximation quality. The resulting evidence estimates can be biased, but variational methods can be computationally efficient for large datasets or complex models.

5 Computational and Practical Considerations

Evidence estimation involves both numerical stability and statistical reliability. Practical success depends on diagnostics, tuning, and careful interpretation.

5.1 High-dimensional challenges

As parameter dimension grows, likelihood and posterior structure can become complex, and the prior mass that meaningfully contributes to evidence can occupy only a tiny fraction of the parameter space. This makes sampling-based estimators difficult: samples may miss relevant regions, increasing variance or introducing bias.

Hierarchical Bayesian models and models with latent variables often fall into this category, where both posterior dependence structure and multimodality can complicate computation.

5.2 Stability and variance of estimators

Many evidence estimators have estimands that are sensitive to tail behavior. For instance, importance sampling can produce extreme weights when the likelihood is high in regions rarely sampled. This yields large variance and unstable estimates. Bridge sampling and annealed methods mitigate this by improving overlap or using intermediate distributions.

Practitioners typically monitor effective sample size, convergence indicators, and reproducibility across multiple runs to assess stability.

5.3 Choice of priors and hyperparameters

Since evidence integrates over the prior, hyperparameters controlling prior scale, sparsity, or smoothness can shift model rankings. Using overly diffuse priors may effectively penalize the model by spreading prior mass over parameter regions that do not fit the data well. Conversely, overly concentrated priors can overstate support.

A practical strategy is to select priors informed by domain knowledge and then evaluate sensitivity by varying hyperparameters within reasonable bounds.

5.4 Diagnostics and uncertainty quantification

Because evidence estimates can be noisy, it is important to quantify uncertainty. Depending on the method, uncertainty may come from:

  • Monte Carlo standard errors,
  • variability across independent runs,
  • posterior predictive checks tied to model calibration,
  • or diagnostic criteria specific to the algorithm (e.g., nested sampling convergence metrics).

Good practice avoids treating a single computed evidence number as exact, especially when model differences are small relative to estimator uncertainty.

6 Asymptotic Approximations

When sample size is large or models are well-behaved, evidence can be approximated using analytic expansions. These approximations are useful for speed and intuition, even if they are not exact for finite data.

6.1 Laplace approximation

The Laplace method approximates the evidence integral by expanding the log joint density around its dominant mode. For regular models, the posterior becomes approximately Gaussian near the maximum a posteriori (MAP) or maximum likelihood estimate (depending on prior influence). The approximation takes a closed form involving:

  • the curvature (Hessian) at the mode,
  • and the prior density evaluated at the mode.

Laplace approximations are common for models where finding the mode and computing curvature are feasible.

6.2 BIC and large-sample behavior

The Bayesian Information Criterion (BIC) is an asymptotic approximation to \(-2\log p(y\mid M)\) under regularity assumptions. It can be written as \[ \mathrm{BIC} \approx -2\log p(y\mid \hat{\theta}, M) + k\log n, \] where \(k\) is the number of free parameters and \(n\) is sample size. BIC often performs similarly to evidence-based selection because both incorporate a complexity penalty that grows with \(n\).

However, BIC depends on assumptions such as identifiability and regularity; when these fail, BIC can misrepresent true evidence.

6.3 Other asymptotic expansions

Beyond Laplace and BIC, more refined approximations exist, including expansions that account for non-Gaussian posterior shapes, boundary effects, or singular models. Such methods can better track evidence when standard regularity conditions do not hold, though they may require more mathematical machinery and stronger modeling assumptions.

7 Applications and Case Studies

Marginal likelihood underlies a variety of model-selection tasks, particularly when comparing distinct modeling choices or structural variants.

7.1 Model selection in regression frameworks

In regression, marginal likelihood can compare models with different predictor sets, different error distributions (e.g., Gaussian vs heavy-tailed), or different prior structures (e.g., variable selection priors). Evidence favors models that explain the data without wasting prior mass on overly flexible parameterizations that are not supported by the observations.

In practice, evidence-based regression selection can be paired with careful prior specification, especially for shrinkage and regularization priors that strongly influence how much complexity the model can express.

7.2 Hierarchical Bayesian models

Hierarchical models introduce group-level parameters and pooling across groups. Evidence provides a principled way to decide between competing hierarchical structures, such as whether group-specific effects should be included, how strongly they should shrink toward a common mean, or which hyperprior family better matches the data.

Because hierarchical models often have more parameters and latent structure, evidence estimation typically relies on approximate algorithms, and sensitivity to priors can be more pronounced.

7.3 Mixture models and latent variable models

Mixture and latent variable models often exhibit multimodal posteriors and identifiability issues. Marginal likelihood can compare mixture cardinalities (e.g., number of components) or competing latent structures. Evidence naturally penalizes overly complex mixtures when extra components are unnecessary, though the computation may be challenging.

In these models, estimation techniques like bridge sampling, annealed methods, or nested sampling are frequently employed to stabilize evidence calculations.

8 Common Pitfalls

Several recurring mistakes can distort evidence-based conclusions.

8.1 Confusing evidence with likelihood

A frequent error is to treat the likelihood at a point (such as the MAP estimate) as if it were the marginal likelihood. The evidence averages the likelihood over parameters using the prior, while the likelihood conditions on a fixed parameter value. As a result, evidence and likelihood-based comparisons can disagree, particularly when priors and posterior uncertainty are substantial.

8.2 Mis-specified priors

Because marginal likelihood integrates over the prior, an inappropriate prior can dominate the result. Mis-specified priors may:

  • understate plausible parameter values,
  • over-diffuse mass into irrelevant regions,
  • or create numerical difficulties (e.g., extremely heavy tails).

Even when the likelihood fit is good, a poor prior can reduce evidence or distort rankings.

8.3 Overfitting vs evidence-driven selection

Evidence-based selection is often described as resisting overfitting due to Occam-like penalties. However, evidence does not guarantee protection against overfitting in every situation. If priors are chosen to be too permissive or if evidence estimation is noisy, the resulting model ranking may reflect computational artifacts or prior-driven behavior rather than genuine predictive structure.

Robust practice includes checking estimator uncertainty and validating selected models through predictive performance measures.

Marginal likelihood appears in multiple closely related quantities. Understanding these variants helps avoid confusion.

9.1 Conditional vs unconditional marginal likelihood

Sometimes evidence is computed conditional on auxiliary variables or subsets of data. For instance, one may compute \(p(y\mid x, M)\) where \(x\) are covariates, treating them as given. In other contexts, one marginalizes over both parameters and latent variables, producing different but related evidence forms:

  • unconditional evidence: integrates only parameter uncertainty given the full observed dataset,
  • conditional evidence: conditions on some observed variables while marginalizing others.

9.2 Integrated likelihood and evidence

The term integrated likelihood is often used synonymously with marginal likelihood, emphasizing that likelihood has been integrated over parameters. In Bayesian settings, “evidence” highlights its role in updating model probabilities and normalizing Bayes’ rule at the model level.

9.3 Marginal likelihood vs marginal posterior

The marginal posterior typically refers to the posterior distribution of a subset of parameters or latent variables, obtained by integrating out the rest: \[ p(\theta_i\mid y,M)=\int p(\theta\mid y,M)\,d\theta_{-i}. \] This differs from evidence, which integrates the likelihood over the prior, not the posterior. Both involve marginalization, but they summarize different stages of the Bayesian update and serve different roles in computation and interpretation.

10 Summary and Key Takeaways

Marginal likelihood provides a single quantity that averages how plausible the data are under all parameter values supported by a model’s prior. It supports principled Bayesian model comparison, balancing fit and complexity in a way that is sensitive to prior specification and computational approximation quality.

10.1 When to use marginal likelihood

Marginal likelihood is especially useful when:

  • comparing non-nested models or different likelihood families,
  • selecting among competing model structures (e.g., component counts in mixtures),
  • performing fully Bayesian model comparison rather than relying on ad hoc penalties,
  • evaluating evidence with Bayes factors or Bayesian model averaging.

It is less convenient when computation is too expensive or when evidence estimates are too uncertain to support clear model ranking.

10.2 Interpretation checklist for evidence-based decisions

A practical checklist for interpreting evidence includes:

  • confirm that evidence (not pointwise likelihood) is being compared,
  • evaluate sensitivity to prior and hyperparameter choices,
  • inspect estimator diagnostics and uncertainty (especially for Monte Carlo approximations),
  • compare models using consistent data splits and definitions,
  • validate selected models using predictive checks or held-out evaluation when feasible.

Used carefully, marginal likelihood offers an interpretable bridge between Bayesian priors, data fit, and model complexity.