1 Definition

A Bayes factor is a statistic used to compare two competing probabilistic models by measuring how well each predicts the observed data. It is a central tool in Bayesian model comparison and is defined as the ratio of the models’ marginal likelihoods. Unlike methods that rely only on point estimates or asymptotic significance, the Bayes factor evaluates the full model predictions after averaging over parameter uncertainty.

Bayes factors are often used when the goal is to quantify relative support for one model versus another. They can compare simple hypotheses, nested models, or more general models with different parameter structures. Because they incorporate prior assumptions as well as observed evidence, their interpretation is inherently Bayesian.

1.1 Mathematical expression

For two models, \(M_1\) and \(M_0\), and observed data \(D\), the Bayes factor in favor of \(M_1\) over \(M_0\) is

\[ BF_{10} = \frac{p(D \mid M_1)}{p(D \mid M_0)}. \]

Each term \(p(D \mid M)\) is the marginal likelihood of the data under model \(M\). If \(BF_{10} > 1\), the data are more predictive under \(M_1\); if \(BF_{10} < 1\), the data favor \(M_0\).

The Bayes factor is dimensionless and can be reported on a natural scale or transformed to logarithms for convenience, especially when values are very large or very small.

1.2 Interpretation as evidence ratio

The Bayes factor is often described as an evidence ratio. A value of 10, for example, means that the observed data are 10 times as probable under one model as under the other, after accounting for each model’s parameter uncertainty. This does not mean that the probability that the model is true is 10 times larger; rather, it compares predictive adequacy for the specific data at hand.

Because the factor compares averaged predictive performance, it balances fit and complexity. A model with many adjustable parameters may fit the data well, but if that flexibility is not justified by the prior predictive distribution, the marginal likelihood can be reduced.

1.3 Relation to posterior odds

Bayes factors are directly connected to posterior odds through Bayes’ theorem:

\[ \frac{p(M_1 \mid D)}{p(M_0 \mid D)} = BF_{10} \times \frac{p(M_1)}{p(M_0)}. \]

Here, the posterior odds equal the Bayes factor multiplied by the prior odds. This identity makes the Bayes factor a mechanism for updating beliefs about competing models. If the prior odds are neutral, the posterior odds are determined entirely by the evidence ratio.

2 Derivation

The Bayes factor arises from integrating each model’s likelihood over its parameter space with respect to the prior distribution. This process converts a parameterized statistical model into a single predictive quantity. The result is a comparison between the models’ overall compatibility with the data rather than a comparison at one selected parameter value.

2.1 Marginal likelihood

For a model \(M\) with parameter vector \(\theta\), the marginal likelihood is

\[ p(D \mid M) = \int p(D \mid \theta, M)\, p(\theta \mid M)\, d\theta. \]

The likelihood \(p(D \mid \theta, M)\) measures data fit for each parameter value, while the prior \(p(\theta \mid M)\) specifies how plausible those values are before seeing the data. The integral averages the likelihood across all parameter values allowed by the model.

This quantity is also called the model evidence in Bayesian statistics. It depends on both the data and the prior distribution, which is why model comparison using Bayes factors is sensitive to prior choice.

2.2 Integration over parameters

The integral defining the marginal likelihood can be understood as a weighted average of the model’s predictive performance. Regions of parameter space with high likelihood contribute substantially only if they also have non-negligible prior mass. As a result, a model is rewarded for fitting the data well only when that fit is achieved without requiring extreme or highly specific parameter values.

This averaging process is one reason Bayes factors naturally penalize overly flexible models. A large parameter space can dilute predictive mass unless the data strongly concentrate support in favorable regions.

2.3 Discrete versus continuous models

If a model has discrete parameters or finitely many states, the marginal likelihood becomes a sum rather than an integral:

\[ p(D \mid M) = \sum_{\theta} p(D \mid \theta, M)\, p(\theta \mid M). \]

For continuous parameters, integration is used instead. In both cases, the same principle applies: the model’s evidence is obtained by averaging its likelihood over the prior distribution. The distinction matters mainly for computation, not for the conceptual definition of the Bayes factor.

3 Computation

Computing Bayes factors can be straightforward in simple cases and difficult in complex ones. Exact formulas are available for some conjugate models, but many practical applications require numerical approximation. The challenge is usually the marginal likelihood, which may involve high-dimensional integration.

3.1 Closed-form calculation

In conjugate Bayesian models, the marginal likelihood can often be derived in closed form. This occurs when the prior and likelihood belong to families that make the integral analytically tractable. Examples include some binomial, normal, and multinomial settings with standard priors.

Closed-form expressions are valuable because they are exact and computationally efficient. They also make it easier to study how the Bayes factor changes with sample size, prior scale, and model complexity.

3.2 Numerical integration

When no closed-form solution exists, numerical integration may be used. This approach directly approximates the integral defining the marginal likelihood. It can work well for low-dimensional parameter spaces, but its cost rises rapidly as dimension increases.

Quadrature methods and related deterministic techniques are typically limited to simpler models. In higher dimensions, they may become impractical because the number of evaluation points needed for adequate precision grows quickly.

3.3 Monte Carlo methods

Monte Carlo approaches estimate the marginal likelihood using random samples from a relevant distribution. They are especially useful in moderate to high dimensions, where direct integration is difficult. Their accuracy depends on sampling efficiency and on how well the proposal or sampling distribution matches the target quantity.

3.3.1 Importance sampling

Importance sampling estimates the marginal likelihood by drawing samples from a proposal distribution and reweighting them to account for the target prior or posterior. When the proposal distribution is close to the target, the method can be efficient and accurate.

However, importance sampling can be unstable if the proposal has lighter tails than the target or misses important regions of parameter space. In such cases, a few samples may dominate the estimate, increasing variance.

3.3.2 Markov chain Monte Carlo

Markov chain Monte Carlo methods generate dependent samples from the posterior distribution, which can then be used indirectly to estimate the marginal likelihood. These methods are highly flexible and widely used in Bayesian analysis.

Because MCMC samples from the posterior rather than directly from the evidence integral, additional techniques are usually needed to convert the output into a Bayes factor estimate. Common strategies include bridge sampling, thermodynamic integration, and related estimators.

3.4 Approximation methods

Approximate techniques are often used when exact evaluation is unavailable and Monte Carlo computation is expensive. These methods trade some precision for speed and convenience. They are especially common in large-sample settings or in model classes where asymptotic approximations are reasonable.

3.4.1 Laplace approximation

The Laplace approximation estimates the marginal likelihood by approximating the posterior distribution near its mode with a Gaussian distribution. This transforms the integral into a form that is easier to evaluate analytically.

The method is often accurate when the posterior is unimodal and reasonably close to normal. Its performance may decline for skewed, multimodal, or strongly constrained posteriors.

3.4.2 BIC-based approximations

The Bayesian information criterion is sometimes used as an approximation to the Bayes factor. Under certain regularity conditions and large sample sizes, differences in BIC values can approximate log Bayes factors. This connection makes BIC attractive as a quick model comparison tool.

The approximation is most reliable when models are regular and sample sizes are large. It is less dependable for small samples, singular models, or situations with informative priors that differ substantially from the assumptions underlying BIC.

4 Properties

Bayes factors have several notable mathematical and inferential properties. They are reciprocal, prior-sensitive, and naturally incorporate a penalty for model complexity. These features distinguish them from methods based solely on likelihood maximization or significance thresholds.

4.1 Symmetry and reciprocity

Bayes factors are reciprocal by construction:

\[ BF_{10} = \frac{1}{BF_{01}}. \]

This symmetry ensures that switching the order of the two models simply inverts the ratio. The property is useful for interpretation because evidence for one model is equivalently evidence against the other.

4.2 Sensitivity to prior specification

Because the marginal likelihood averages over the prior, the Bayes factor can change substantially when the prior distribution changes. Priors that spread probability mass widely may lower the evidence for a model by assigning weight to parameter values that fit the data poorly. More concentrated priors can have the opposite effect, although they may be overly restrictive.

This sensitivity means that Bayes factors should be reported with careful attention to prior choices. Sensitivity analysis is often recommended when priors are not strongly justified by substantive knowledge.

4.3 Occam's razor effect

Bayes factors typically embody an Occam’s razor effect: among models with similar fit, the simpler one is favored. This occurs because a simpler model concentrates prior predictive mass in a smaller set of outcomes, making it less costly, in evidence terms, to explain the observed data.

The effect is not an arbitrary penalty. It arises from the averaging process in the marginal likelihood, which rewards models that make sharper predictions without losing empirical adequacy.

4.4 Sample size behavior

As sample size increases, Bayes factors often become more decisive, provided the models differ in meaningful ways. If one model is closer to the data-generating process, the evidence ratio tends to move toward that model. In many regular settings, the Bayes factor grows exponentially with the amount of information available.

At the same time, large samples can magnify the influence of prior misspecification or model inadequacy. When both models are imperfect, the Bayes factor reflects relative rather than absolute fit.

5 Interpretation and scale

Interpreting Bayes factors requires attention to magnitude, context, and prior assumptions. A numerical value alone does not always capture practical importance, especially when the competing models are close in predictive ability. Researchers often use rough verbal scales, but these are conventions rather than universal rules.

5.1 Evidential thresholds

Some authors propose thresholds for categorizing evidence, such as weak, moderate, strong, or very strong support. These thresholds are intended as guides, not rigid decision rules. A Bayes factor near 1 suggests that the data do not distinguish clearly between the models.

Thresholds can be useful for communication, but they may oversimplify inference. The meaning of a given value depends on the scientific context, the models being compared, and the consequences of model choice.

5.2 Common verbal descriptors

Common descriptions include phrases such as “anecdotal evidence,” “substantial evidence,” and “decisive evidence.” These labels are used to translate numerical ratios into more accessible language. They are often associated with informal scales in Bayesian practice.

Because the same numeric Bayes factor may be viewed differently across disciplines, such verbal descriptors should be interpreted cautiously. They summarize strength of evidence, not certainty.

5.3 Comparison with p-values

Bayes factors differ fundamentally from p-values. A p-value measures how surprising the data are under a null hypothesis, assuming that hypothesis is true, while a Bayes factor compares predictive support between two models. The Bayes factor therefore provides a direct comparison of models, whereas the p-value is not a model probability.

The two quantities can sometimes point in different directions because they answer different questions. Bayes factors incorporate prior information and relative model complexity, while p-values are based on sampling distributions under a specified null model.

6 Model comparison applications

Bayes factors are widely used in Bayesian workflows where several models are under consideration. They can guide hypothesis testing, variable selection, averaging across models, and prediction-oriented analysis. Their flexibility makes them suitable for both confirmatory and exploratory settings.

6.1 Hypothesis testing

In hypothesis testing, Bayes factors compare a null model to one or more alternatives. This approach can be used for point nulls, order restrictions, or nested hypotheses. The resulting evidence ratio indicates which hypothesis is better supported by the observed data.

Unlike classical tests, Bayes factor analysis can support evidence for the null model rather than merely fail to reject it. This feature is particularly useful when absence of an effect is itself a substantive claim.

6.2 Variable selection

Bayes factors are often used to compare regression models with different predictor sets. Each candidate model represents a different subset of variables, and the evidence ratio identifies which combination provides the best balance of fit and parsimony.

This application is common in exploratory modeling, where many candidate predictors may be considered. The Bayes factor can favor a smaller model if additional variables do not improve predictive support enough to justify their complexity.

6.3 Model averaging

Instead of choosing a single best model, analysts may use Bayes factors to compute posterior model probabilities and then average predictions across models. This approach accounts for model uncertainty and can produce more stable inference than relying on one selected specification.

Model averaging is especially helpful when several models receive similar support. In that case, the Bayes factor provides weights for combining their contributions in a principled way.

6.4 Predictive assessment

Bayes factors are closely tied to predictive evaluation because they compare how well models forecast observed data. They are often used when predictive adequacy matters more than parameter estimation alone. In this sense, the Bayes factor can be viewed as a score for model-based prediction under uncertainty.

Predictive use is strongest when the models are genuinely competing explanations rather than merely different parameterizations of the same process. It is less informative when models are nearly identical in their implications.

7 Limitations

Despite their advantages, Bayes factors have practical and conceptual limitations. These include dependence on priors, computational burden, and difficulties arising from improper prior distributions. Careful model specification is therefore essential.

7.1 Prior dependence

The Bayes factor can change notably with different prior choices. This dependence is not a flaw in the formula itself, but it means that the result reflects both data and prior assumptions. When priors are weakly justified, conclusions may be unstable.

Researchers often address this issue by reporting sensitivity analyses, using priors grounded in substantive knowledge, or examining whether conclusions hold across reasonable prior ranges.

7.2 Computational complexity

For complex models, computing the marginal likelihood may be expensive or numerically delicate. High-dimensional integrals, multimodal posteriors, and hierarchical structures can all make estimation difficult. Approximate methods may help, but they can introduce additional uncertainty.

The computational challenge is one reason Bayes factors are sometimes avoided in favor of posterior predictive checks or information criteria in very large or complex models.

7.3 Improper priors

Improper priors do not integrate to one, and they can make marginal likelihoods undefined. Since the Bayes factor requires a proper prior for each model, using improper priors can invalidate the comparison. This issue is especially important in objective Bayesian analysis, where noninformative priors are sometimes attractive.

Special methods exist for some cases, but they must be applied carefully. In general, a proper prior is needed to ensure a well-defined Bayes factor.

7.4 Robustness concerns

Because the Bayes factor reflects both modeling assumptions and numerical approximation, its stability should be checked when possible. Small changes in prior scale, model parameterization, or computation method can sometimes alter the outcome. Robustness analysis is therefore a useful part of responsible application.

When conclusions are highly sensitive, the evidence should be interpreted conservatively. In such settings, a Bayes factor may indicate preference but not strong decisiveness.

Bayes factors are part of a broader Bayesian framework for inference and decision-making. Several related concepts help clarify their meaning and role. These include posterior probability, likelihood ratios, information criteria, and Bayesian evidence.

8.1 Posterior probability

Posterior probability is the probability assigned to a model after observing data and applying Bayes’ theorem. Bayes factors contribute to posterior probabilities by updating prior odds. While the Bayes factor measures relative evidence, posterior probability combines that evidence with prior belief.

This distinction matters because a strong Bayes factor does not necessarily produce a high posterior probability if the prior odds are very unfavorable. Conversely, a modest Bayes factor may still yield a substantial posterior probability under favorable prior odds.

8.2 Likelihood ratio

A likelihood ratio compares the likelihoods of two models at fixed parameter values. It is similar in form to a Bayes factor but does not integrate over parameter uncertainty. The Bayes factor can be viewed as a prior-weighted version of model comparison that averages over possible parameter values.

This distinction makes the Bayes factor more comprehensive in Bayesian settings. It evaluates each model as a whole rather than at a single best-fitting parameter point.

8.3 Information criteria

Information criteria such as AIC and BIC are alternative model-selection tools. Like Bayes factors, they trade off fit against complexity, but they are derived from different principles. AIC emphasizes predictive accuracy, while BIC has a closer asymptotic relationship to Bayes factors under specific conditions.

These criteria are often easier to compute, but they do not directly provide an evidence ratio. As a result, their interpretation differs from the Bayes factor’s explicit comparison of model support.

8.4 Bayesian evidence

Bayesian evidence is another name for marginal likelihood. It is the quantity used in the numerator and denominator of a Bayes factor. In many contexts, “evidence” refers to the same object as model evidence, though the wording may vary by discipline.

Because the Bayes factor is a ratio of evidences, understanding marginal likelihood is essential to understanding Bayesian model comparison.

9 History

The Bayes factor developed from early work in probability and Bayesian reasoning, then became a formal tool in modern statistical practice. Its evolution reflects broader changes in how statisticians think about evidence, uncertainty, and computation.

9.1 Bayes and early Bayesian inference

The conceptual roots of Bayes factors lie in the ideas associated with Thomas Bayes and later contributors to Bayesian inference. Early Bayesian reasoning emphasized updating beliefs with new information, which naturally leads to comparing how well different hypotheses explain data.

Although the modern term came later, the underlying logic was present in early probability-based approaches to inference. Model comparison by relative predictive support gradually became a recognizable theme within Bayesian analysis.

9.2 Development in modern statistics

In the twentieth century, Bayesian methods were refined into a systematic framework for inference. As statistical models became more elaborate, the need for principled comparison tools increased. The Bayes factor emerged as a standard way to evaluate competing models while accounting for uncertainty in parameters.

Its use broadened in fields such as psychology, biostatistics, econometrics, and signal detection. The method gained particular prominence where one wanted to compare evidence for alternative scientific explanations rather than test a single null hypothesis.

9.3 Contributions to computational Bayesian methods

The practical use of Bayes factors expanded with advances in computation. As numerical integration, Monte Carlo simulation, and Markov chain Monte Carlo became more accessible, estimating marginal likelihoods became feasible for increasingly complex models. This computational progress made Bayes factors usable in real-world applications beyond simple textbook cases.

Modern work on bridge sampling, thermodynamic integration, and related estimators has further improved the calculation of model evidence. These developments continue to shape how Bayes factors are applied in contemporary Bayesian statistics.

</INTERNAL_LINK_CANDIDATES> Marginal likelihood (the data probability under a model after averaging over its parameters) Posterior odds (the updated odds between models after observing data) Prior odds (the starting odds between competing models before data) Posterior probability (the probability of a model after data are observed) Likelihood ratio (the ratio of likelihoods for two models at fixed parameter values) Bayesian model comparison (the process of comparing models using Bayesian evidence) Bayesian inference (a framework for updating beliefs with data) Model evidence (another name for marginal likelihood) Hypothesis testing (formal comparison of competing hypotheses) Variable selection (choosing predictors in a regression model) Model averaging (combining predictions from multiple models using their weights) Predictive assessment (evaluating how well a model forecasts data) Laplace approximation (a Gaussian-based approximation to integrals) BIC (an information criterion sometimes used to approximate Bayes factors) Importance sampling (a Monte Carlo method for estimating integrals) Markov chain Monte Carlo (simulation methods for sampling from posterior distributions) Bridge sampling (a method for estimating marginal likelihoods from posterior draws) Thermodynamic integration (an approach for computing model evidence) Improper prior (a prior distribution that does not integrate to one) Occam's razor (the principle favoring simpler explanations when fit is similar)