1 Definition and basic formulation

1.1 Likelihood, prior, and marginalization

In a statistical model with observed data \(y\), unknown parameters \(\theta\), and a chosen prior distribution \(p(\theta)\), the likelihood \(p(y\mid \theta)\) describes how probable the data are for each parameter value. Evidence, also known as the marginal likelihood, combines these likelihood values across the entire parameter space by averaging them under the prior. This averaging is called marginalization: instead of treating \(\theta\) as fixed but unknown, the computation integrates (or sums) over \(\theta\) to obtain a single quantity that depends only on the data and the model structure.

1.2 Mathematical expression for marginal likelihood

Let \(\mathcal{M}\) denote a model (including likelihood form and prior). The marginal likelihood, also written \(p(y\mid \mathcal{M})\), is defined as \[ p(y\mid \mathcal{M})=\int p(y\mid \theta,\mathcal{M})\,p(\theta\mid \mathcal{M})\,d\theta. \] This is the probability of the observed data under the model, after accounting for uncertainty in \(\theta\). In practice, computations often use log-evidence to avoid numerical underflow: \[ \log p(y\mid \mathcal{M}). \]

1.3 Discrete vs. continuous parameter spaces

If parameters are discrete, the integral becomes a sum: \[ p(y\mid \mathcal{M})=\sum_{\theta} p(y\mid \theta,\mathcal{M})\,p(\theta\mid \mathcal{M}). \] If parameters are continuous, the integral is typically high-dimensional in realistic models. The computational difficulty of evidence estimation is therefore tightly linked to the dimensionality and geometry of the parameter space.

2 Relationship to Bayesian inference

2.1 Evidence as a normalizing constant

Bayesian inference updates a prior \(p(\theta\mid\mathcal{M})\) to a posterior \(p(\theta\mid y,\mathcal{M})\) via Bayes’ rule: \[ p(\theta\mid y,\mathcal{M})=\frac{p(y\mid \theta,\mathcal{M})\,p(\theta\mid \mathcal{M})}{p(y\mid \mathcal{M})}. \] In this identity, the evidence \(p(y\mid\mathcal{M})\) serves as the normalizing constant that ensures the posterior integrates to one.

2.2 Posterior distribution perspective

From the posterior perspective, evidence determines how strongly the prior-weighted likelihood supports the model overall. Informally, if \(p(y\mid\theta,\mathcal{M})\) is large across a substantial region of high prior density, the evidence tends to be large. Conversely, if good likelihood occurs only in narrow prior regions or in low-prior-density areas, the evidence may be small due to dilution when averaging over \(\theta\).

2.3 Model comparison via Bayes factors

When comparing two models \(\mathcal{M}_1\) and \(\mathcal{M}_2\), Bayesian model selection often uses the Bayes factor: \[ \mathrm{BF}_{12}=\frac{p(y\mid\mathcal{M}_1)}{p(y\mid\mathcal{M}_2)}. \] A Bayes factor greater than one indicates that \(\mathcal{M}_1\) yields higher marginal likelihood for the observed data, while values less than one favor \(\mathcal{M}_2\). If model priors \(p(\mathcal{M}_1)\) and \(p(\mathcal{M}_2)\) are included, posterior model probabilities follow directly from Bayes’ rule at the model level.

3 Interpretation and intuition

3.1 Occam’s razor and effective model complexity

Evidence embodies a trade-off between fit and complexity. Models with flexible parameterizations can often achieve high likelihood, but that does not guarantee large evidence. Because evidence averages likelihood over the prior volume, parameter regions that achieve good fit may occupy only a small portion of the prior mass. This penalizes overly flexible models unless the data concentrate posterior mass into a sufficiently large region relative to the prior. The result is frequently described as an Occam’s razor effect: simpler explanations receive preference unless complex ones are strongly supported by the data.

3.2 Predictive interpretation (“data given model”)

Evidence can be viewed as the model’s predictive performance for the particular dataset \(y\), before seeing \(y\). In Bayesian terms, it is the probability of observing \(y\) when parameters are drawn from the prior and then used to generate data. This makes evidence a natural quantity for comparing models by their implied predictions.

3.3 Sensitivity to prior choices

Because evidence integrates the likelihood against the prior, it depends on the prior distribution not only through its normalization but also through where prior mass lies. Two models with identical likelihood forms but different priors can yield markedly different evidences. This sensitivity is most pronounced when the data are weak relative to the prior or when priors differ in effective support, scale, or tail behavior.

4 Special cases and illustrative models

4.1 Conjugate priors and closed-form evidence

For certain model families, conjugate priors allow the marginal likelihood integral to be evaluated analytically. Conjugacy typically ensures that the posterior distribution belongs to the same family as the prior, making it possible to derive closed-form expressions for \(p(y\mid \mathcal{M})\). These cases are widely used for teaching because they clarify how evidence balances prior volume and data fit.

4.2 Linear-Gaussian models and analytic marginal likelihood

In linear-Gaussian settings—such as Bayesian linear regression with Gaussian noise and Gaussian priors on coefficients—the evidence often has an analytic form. The marginalization over coefficients yields a multivariate Gaussian distribution for the data (or for transformed residuals). As a result, evidence can be computed using matrix operations, including determinants and quadratic forms.

4.3 Hierarchical models and nested marginalization

Hierarchical models introduce additional latent variables and hyperparameters. Evidence then requires marginalizing over all unknown quantities, often via nested integrals. For example, one may integrate over local parameters given hyperparameters, then integrate over hyperparameters using their priors. Although the overall expression can be written as a single marginal likelihood, practical derivations frequently perform marginalization in stages to simplify computation or to expose conditional conjugacy.

5 Computational approaches

5.1 Numerical integration and quadrature (low-dimensional cases)

When the parameter space is low-dimensional, evidence can sometimes be approximated through deterministic numerical integration or quadrature. Grid-based methods or adaptive quadrature evaluate the integrand at selected parameter values and approximate the integral. This approach can become infeasible as dimensionality grows because the number of required evaluations increases rapidly.

5.2 Laplace approximation and asymptotic methods

The Laplace approximation approximates the posterior (or the integrand of evidence) near its dominant mode. By expanding the log joint density to second order around the maximum a posteriori or maximum likelihood region, evidence can be approximated with quantities involving the mode location and the curvature (Hessian) at that point. Accuracy is usually highest when the posterior is sharply peaked and approximately Gaussian in the relevant region.

5.3 Monte Carlo estimation (general overview)

Monte Carlo methods approximate the evidence integral by sampling parameter values according to some proposal distribution and averaging weighted likelihood terms. Because direct sampling from the prior may waste effort in regions where the likelihood is informative, many Monte Carlo estimators incorporate importance weights or other strategies to improve variance and computational efficiency.

5.4 Importance sampling for evidence

Importance sampling replaces the prior integral with an expectation under a proposal distribution \(q(\theta)\): \[ p(y\mid \mathcal{M})=\mathbb{E}_{q(\theta)}\left[\frac{p(y\mid \theta,\mathcal{M})\,p(\theta\mid\mathcal{M})}{q(\theta)}\right]. \] The quality of the estimate depends on how well \(q(\theta)\) covers regions where the integrand is large. Poor proposals lead to high-variance weights and unstable estimates.

Bridge sampling constructs an estimator by “bridging” between distributions, often between the posterior and a reference distribution. The method can reduce variance relative to naive importance sampling by optimizing how samples contribute across the probability mass of interest. Variants include estimators that use both posterior and proposal samples, typically requiring additional computation but offering improved reliability.

5.6 Nested sampling

Nested sampling estimates evidence by transforming the multi-dimensional evidence integral into a one-dimensional integral over decreasing likelihood thresholds. It works by repeatedly sampling points within constrained regions of increasing likelihood and tracking how prior mass shrinks as constraints tighten. This method is often effective for multimodal posteriors but can still require careful tuning to ensure adequate exploration.

5.7 Variational and bound-based approximations

Variational methods approximate the posterior with a simpler distribution \(q(\theta)\) and optimize an objective that yields a lower bound on log evidence (e.g., the evidence lower bound). Such approaches provide scalable approximations but may understate evidence if the variational family cannot represent the true posterior well. Bound-based estimators are most useful when exact marginal likelihood is too costly.

6 Diagnostics and practical considerations

6.1 Numerical stability (log-space computation)

Likelihoods can be extremely small, especially in moderate or large datasets, leading to underflow if computed in raw probability space. Evidence calculations typically use log-space transformations, such as computing \(\log p(y\mid\theta)+\log p(\theta)-\log q(\theta)\) and applying log-sum-exp style stabilization. Stable implementation is crucial for trustworthy estimates.

6.2 Tuning and convergence for stochastic estimators

Stochastic estimators (e.g., importance sampling, bridge sampling, nested sampling) require tuning choices such as proposal distributions, number of samples, and stopping criteria. Convergence diagnostics may include monitoring effective sample size, weight distributions, or changes in evidence estimates across iterations. Without sufficient sampling, the estimate can be dominated by rare events, producing misleading results.

6.3 Assessing estimator variance and reliability

Because evidence estimates can have substantial Monte Carlo error, reporting uncertainty or estimating variance is important. Methods include bootstrap-style resampling, multiple independent runs, or analytical variance approximations when available. Reliable comparisons between models require that the difference in evidences exceeds the estimation noise.

6.4 Prior support and truncation effects

Evidence can be sensitive to how priors are bounded or truncated. If a prior has compact support or is numerically implemented with finite bounds, the evidence integral effectively changes. Diagnostics therefore often consider whether the posterior mass lies well within the intended prior support, and whether truncation artifacts could bias the evidence.

7 Evidence in modern workflows

7.1 Model selection pipelines

In applied Bayesian workflows, evidence is used to rank candidate models, sometimes as part of automated search over model structures (e.g., selecting among likelihood families, transformations, or regularization forms). Practical pipelines typically combine evidence estimation with safeguards like variance checks and sensitivity analysis to priors.

7.2 Bayesian averaging and predictive model choice

Instead of selecting a single model, Bayesian model averaging uses posterior model probabilities, which depend on evidences. The resulting predictive distribution averages over models, weighted by how well each model explains the observed data. This can reduce the risk of committing to an incorrect model form when uncertainty about model structure remains.

7.3 Comparing nested vs. non-nested models

Evidence naturally supports both nested and non-nested comparisons. Nested models differ by constraint or parameter specialization; non-nested models differ more fundamentally in structure. In nested settings, evidence still provides a penalty for unnecessary flexibility, but the computational cost may differ because the parameter spaces and priors may require different treatment.

8 Connections to other concepts

8.1 Evidence vs. likelihood ratio tests

Likelihood ratio tests compare models using the ratio of maximized likelihoods, focusing on fit at parameter estimates. Evidence differs by integrating over parameter uncertainty using the prior. As a result, evidence incorporates both goodness-of-fit and prior volume effects, while likelihood ratio tests do not directly account for how much of the parameter space yields plausible explanations.

8.2 Evidence and cross-validation relationships high level

Cross-validation evaluates predictive performance by repeatedly training on subsets of data and testing on held-out portions. Evidence is also tied to predictive adequacy, but through a prior-weighted probability of the full dataset rather than through resampling-based predictive checks. Under certain regularity conditions and approximations, connections exist between evidence-based criteria and cross-validation behavior, though they are not identical objects.

8.3 Information criteria linkages e.g. approximate correspondences

Information criteria such as AIC or BIC are widely used for approximate model selection. While they are derived from different principles, they can exhibit qualitative relationships to evidence: for example, BIC is often discussed as an approximation to log evidence in large-sample regimes under suitable assumptions. These correspondences can guide intuition about how evidence balances fit and complexity.

9 Common pitfalls and misconceptions

9.1 Confusing evidence with likelihood

A common mistake is to treat evidence as the likelihood evaluated at the best parameters. Evidence is an integral that averages likelihood over the prior, producing a value that reflects both model fit and parameter uncertainty. Consequently, evidence cannot generally be inferred from a single likelihood value.

9.2 Ignoring prior dependence

Because evidence depends on the chosen prior distribution, changing priors can change the ranking of models. Ignoring this dependency can lead to overconfident conclusions. Good practice often includes checking sensitivity to reasonable prior choices or using priors that reflect genuine prior knowledge.

9.3 Using unreliable approximations without checks

Approximations such as Laplace methods, variational bounds, or limited-sample Monte Carlo can fail when posteriors are multi-peaked, heavy-tailed, or poorly captured by the approximation family. Without diagnostics—such as stability checks, variance estimates, or comparison across multiple estimators—evidence estimates may be numerically or statistically unreliable.