Overview: Bayes' theorem is a fundamental principle in probability theory that describes the probability of an event based on prior knowledge of conditions that might be related to the event. Formally, it expresses the conditional probability of an event A given that event B has occurred in terms of the conditional probability of B given A, the prior probability of A, and the marginal probability of B. Named after the Reverend Thomas Bayes (c. 1701–1761), the theorem forms the core of Bayesian inference, a method of statistical reasoning where evidence is used to update the plausibility of a hypothesis.

1 Definition and statement

1.1 Mathematical formula

Bayes' theorem is mathematically stated as

\[ P(A \mid B) = \frac{P(B \mid A) \, P(A)}{P(B)}, \]

where \(P(A \mid B)\) is the conditional probability of event \(A\) given that \(B\) has occurred, \(P(B \mid A)\) is the conditional probability of \(B\) given \(A\), \(P(A)\) is the prior probability of \(A\), and \(P(B)\) is the marginal probability of \(B\). The formula holds provided \(P(B) > 0\).

1.2 Notation and terminology

1.2.1 Prior probability

The prior probability \(P(A)\) represents the initial degree of belief in event \(A\) before any evidence is considered. It encapsulates existing knowledge or assumptions about the event.

1.2.2 Likelihood

The likelihood \(P(B \mid A)\) is the probability of observing evidence \(B\) given that hypothesis \(A\) is true. It quantifies how compatible the observed data are with the hypothesis.

1.2.3 Marginal likelihood

The marginal likelihood \(P(B)\) is the total probability of the evidence \(B\) under all possible hypotheses, often computed using the law of total probability. It serves as a normalizing constant that ensures the posterior probability sums to one.

1.2.4 Posterior probability

The posterior probability \(P(A \mid B)\) is the updated probability of hypothesis \(A\) after taking the evidence \(B\) into account. It combines prior knowledge with the likelihood of the observed data.

2 Derivation

2.1 From the definition of conditional probability

Conditional probability is defined as

\[ P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \qquad P(B \mid A) = \frac{P(A \cap B)}{P(A)}. \]

Solving for \(P(A \cap B)\) in the second equation and substituting into the first yields

\[ P(A \mid B) = \frac{P(B \mid A) P(A)}{P(B)}. \]

2.2 Using the law of total probability

The denominator \(P(B)\) can be expanded using the law of total probability when \(A\) is part of a partition \(\{A_1, A_2, \dots, A_n\}\) of the sample space:

\[ P(B) = \sum_{i} P(B \mid A_i) P(A_i). \]

Substituting this into the formula gives the extended form.

2.3 Alternative forms

2.3.1 Odds form

The odds form expresses the theorem in terms of prior and posterior odds:

\[ \frac{P(A \mid B)}{P(\neg A \mid B)} = \frac{P(A)}{P(\neg A)} \cdot \frac{P(B \mid A)}{P(B \mid \neg A)}. \]

The factor \(\frac{P(B \mid A)}{P(B \mid \neg A)}\) is the Bayes factor, which measures the strength of evidence.

2.3.2 Extended form for multiple events

For a partition \(\{A_1, A_2, \dots, A_n\}\) the posterior probability of a particular \(A_k\) is

\[ P(A_k \mid B) = \frac{P(B \mid A_k) P(A_k)}{\sum_{i=1}^n P(B \mid A_i) P(A_i)}. \]

3 Interpretation

3.1 Bayesian interpretation

In the Bayesian interpretation, probability is treated as a degree of belief, and Bayes’ theorem provides a rational rule for updating beliefs in light of new evidence. This approach is subjective and allows the incorporation of prior information.

3.2 Frequentist interpretation

Frequentists interpret probability as the long‑run relative frequency of events. In this view, Bayes’ theorem is a valid mathematical identity but is not used for subjective updating; instead, inference relies on sampling distributions and likelihood functions.

3.3 Subjective probability and belief updating

Bayesian reasoning explicitly models subjective probability, where different individuals may hold different prior beliefs. The theorem provides a coherent mechanism to revise those beliefs as data accumulate, leading to a posterior that reflects both prior knowledge and empirical evidence.

4.1 Bayes' rule for densities

When dealing with continuous random variables, Bayes’ theorem takes the form of probability density functions:

\[ f_{X \mid Y}(x \mid y) = \frac{f_{Y \mid X}(y \mid x) \, f_X(x)}{f_Y(y)}. \]

This is essential for parameter estimation in continuous models.

4.2 Sequential updating

Bayes’ theorem can be applied iteratively: the posterior from one analysis becomes the prior for the next. This property simplifies the incorporation of new data without recalculating from scratch.

4.3 Relationship to likelihood principle

The likelihood principle states that all evidence in a statistical experiment is contained in the likelihood function. Bayesian inference adheres to this principle because the posterior depends on data only through the likelihood, while priors are chosen separately.

5 Applications

5.1 Statistical inference

5.1.1 Parameter estimation

Bayesian estimation treats unknown parameters as random variables. The posterior distribution of a parameter, given data, is obtained via Bayes’ theorem. Point estimates (e.g., posterior mean) and interval estimates (credible intervals) are derived from this distribution.

5.1.2 Hypothesis testing

Bayesian hypothesis testing compares models by evaluating Bayes factors or posterior probabilities of competing hypotheses, providing a direct measure of evidence in favor of one hypothesis over another.

5.2 Machine learning

5.2.1 Naive Bayes classifiers

Naive Bayes classifiers apply Bayes’ theorem with a strong independence assumption among features. Despite this simplification, they perform well in text classification, spam filtering, and other tasks with high‑dimensional data.

5.2.2 Bayesian networks

A Bayesian network is a graphical model that represents a set of variables and their conditional dependencies via a directed acyclic graph. Inference in such networks relies on repeated application of Bayes’ theorem.

5.3 Medicine

5.3.1 Diagnostic testing

In medical diagnostics, Bayes’ theorem computes the probability that a patient has a disease given a positive test result, using the test’s sensitivity and specificity along with the disease prevalence.

5.3.2 Prevalence and predictive values

The positive predictive value and negative predictive value of a test are direct applications of Bayes’ theorem, relating the prevalence of a condition to test accuracy.

5.4 Jurisprudence and forensic science

Bayes’ theorem is used to evaluate the weight of forensic evidence, such as DNA matches or fingerprint analysis, by comparing the probability of observing the evidence under the prosecution’s hypothesis versus the defense’s hypothesis.

5.5 Other fields

5.5.1 Ecology and biology

Ecologists apply Bayesian methods to estimate population sizes, species distribution, and evolutionary parameters, where prior knowledge from previous studies can be formally incorporated.

5.5.2 Finance and risk analysis

In finance, Bayes’ theorem is used to update beliefs about asset returns, credit risk, and market volatility in response to new economic data.

6 History

6.1 Thomas Bayes and the essay

The theorem first appeared in a posthumous essay by Thomas Bayes, titled *An Essay towards solving a Problem in the Doctrine of Chances* (1763). Bayes, a Presbyterian minister, formulated the problem of inverse probability, though his original work did not state the theorem in its modern form.

6.2 Later developments by Laplace

Pierre‑Simon Laplace independently rediscovered and generalized Bayes’ theorem around 1774. He applied it to celestial mechanics, legal decision‑making, and demography, and derived the first formal statement of the theorem as it is known today.

6.3 20th-century revival

Bayesian methods waned in the 19th and early 20th centuries due to the rise of frequentist statistics. The mid‑20th century saw a revival, driven by the work of Harold Jeffreys, Leonard J. Savage, and others, along with the development of computational tools that made Bayesian analysis practical for complex problems.

7 Computational methods

7.1 Conjugate priors

Conjugate priors are families of probability distributions that, when combined with a given likelihood, yield a posterior distribution in the same family. This analytical tractability simplifies computation, especially in exponential family models.

7.2 Markov chain Monte Carlo

Markov chain Monte Carlo (MCMC) methods, such as the Metropolis‑Hastings algorithm and Gibbs sampling, allow sampling from complex posterior distributions. They are widely used when closed‑form solutions are unavailable.

7.3 Approximate Bayesian computation

Approximate Bayesian computation (ABC) is a simulation‑based technique for models with intractable likelihoods. It estimates the posterior by comparing observed data to simulated data using summary statistics, accepting parameter values that produce close matches.

8 Limitations and common misconceptions

8.1 Base rate fallacy

The base rate fallacy occurs when individuals ignore the prior probability (base rate) of an event and over‑rely on specific evidence. For example, a rare disease with a highly sensitive test can still yield a high false‑positive rate because the disease is uncommon.

8.2 Sensitivity to prior choices

Bayesian results can be sensitive to the choice of prior distribution. Different priors may lead to different conclusions, especially with small sample sizes. This sensitivity is sometimes criticized as introducing subjectivity, though robustness checks and objective priors can mitigate the issue.

8.3 Computational complexity

For high‑dimensional or complex models, Bayesian computation can be demanding. MCMC methods may require extensive tuning and long run times, and convergence diagnostics are needed to ensure reliable results.

9 See also