1 Statement of the theorem

Bayes’ theorem describes how to revise the probability of a hypothesis after observing new evidence. It links the probability of an event before the evidence is known with the probability after the evidence is taken into account. In its simplest form, the theorem shows that a posterior probability can be obtained by combining a prior probability with the likelihood of the observed data.

1.1 Basic formula

For two events \(A\) and \(B\) with nonzero probability of \(B\), Bayes’ theorem is commonly written as

\[ P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}. \]

Here \(P(A \mid B)\) is the probability of \(A\) given \(B\). The formula states that this conditional probability equals the probability of observing \(B\) if \(A\) is true, multiplied by the prior probability of \(A\), and divided by the overall probability of \(B\).

1.2 Conditional probability form

Bayes’ theorem follows directly from the definition of conditional probability. Since

\[ P(A \mid B) = \frac{P(A \cap B)}{P(B)} \quad \text{and} \quad P(B \mid A) = \frac{P(A \cap B)}{P(A)}, \]

the shared joint probability \(P(A \cap B)\) can be rewritten in two ways. Equating these expressions yields the theorem. This form emphasizes that Bayes’ theorem is not a separate rule but a rearrangement of basic probability definitions.

1.3 Equivalent expressions

Bayes’ theorem can be expressed in several equivalent forms depending on the problem being studied. These versions are often more convenient in statistics and applied probability.

1.3.1 Two-event form

For two events \(A\) and \(B\), one may write

\[ P(A \mid B) = \frac{P(B \mid A)P(A)}{P(B)}. \]

This is the standard version used in introductory treatments and simple inference problems.

1.3.2 Multiple-hypothesis form

When several hypotheses \(H_1, H_2, \dots, H_n\) are possible, Bayes’ theorem becomes

\[ P(H_i \mid E) = \frac{P(E \mid H_i)P(H_i)}{\sum_{j=1}^{n} P(E \mid H_j)P(H_j)}, \]

where \(E\) is the evidence. This form is central in model comparison and classification, since it allows the probability of each hypothesis to be updated after observing the same data.

1.4 Interpretation of terms

The theorem separates the information in a structured way. The prior \(P(A)\) represents the initial belief before new data. The likelihood \(P(B \mid A)\) measures how well the evidence fits the hypothesis. The denominator \(P(B)\) normalizes the result so that the final probabilities are consistent. The posterior \(P(A \mid B)\) is the updated belief after the evidence has been observed.

2 Derivation

Bayes’ theorem can be derived in several related ways. The most direct derivation uses the definition of conditional probability, while broader versions rely on the law of total probability and partitions of the sample space.

2.1 From the definition of conditional probability

Starting from the definition,

\[ P(A \mid B) = \frac{P(A \cap B)}{P(B)} \quad \text{and} \quad P(B \mid A) = \frac{P(A \cap B)}{P(A)}, \]

we solve the second equation for \(P(A \cap B)\):

\[ P(A \cap B) = P(B \mid A)P(A). \]

Substituting into the first expression gives

\[ P(A \mid B) = \frac{P(B \mid A)P(A)}{P(B)}. \]

This derivation shows that the theorem is an algebraic consequence of standard probability definitions.

2.2 Using the law of total probability

If events \(H_1, H_2, \dots, H_n\) form a partition of the sample space, then

\[ P(E) = \sum_{j=1}^{n} P(E \mid H_j)P(H_j). \]

Substituting this into Bayes’ formula gives the multiple-hypothesis version:

\[ P(H_i \mid E) = \frac{P(E \mid H_i)P(H_i)}{\sum_{j=1}^{n} P(E \mid H_j)P(H_j)}. \]

This derivation is especially useful when the observed evidence could have been produced by more than one competing explanation.

2.3 General derivation for partitions

For a partition of events, Bayes’ theorem can be used to update each component of the partition separately. The total probability of the evidence is obtained by summing over all mutually exclusive possibilities. Each posterior probability is then proportional to the product of its prior probability and its likelihood. This proportionality is one of the key ideas in Bayesian analysis.

3 Intuitive meaning

Bayes’ theorem is often explained as a rule for learning from evidence. It combines what was believed beforehand with how strongly the new information supports a particular explanation. The result is an updated probability that reflects both sources of information.

3.1 Prior probability

The prior probability is the starting point before the evidence is examined. It may come from past data, long-run frequency, expert judgment, or a simplified model. In practical work, priors can have a substantial effect when data are sparse, while their influence may diminish as more evidence accumulates.

3.2 Likelihood

The likelihood measures how probable the observed evidence would be under a given hypothesis. A hypothesis that makes the data seem plausible receives a higher likelihood. In many applications, likelihood is the main way the evidence distinguishes between competing explanations.

3.3 Posterior probability

The posterior probability is the updated degree of belief after the evidence is taken into account. It combines the prior and likelihood in a single numerical result. In Bayesian reasoning, posterior probabilities are often used as the basis for further inference, since they can become priors for later observations.

3.4 Evidence or marginal probability

The evidence, also called the marginal probability, is the overall probability of observing the data under all relevant hypotheses. It serves as a normalizing factor and ensures that the posterior probabilities sum to one across all possibilities. In model comparison, this quantity also indicates how well the full set of hypotheses explains the observation.

4 Simple examples

Concrete examples help show how Bayes’ theorem works in practice. In each case, the calculation begins with an initial probability and then adjusts it using the observed evidence.

4.1 Medical testing example

Suppose a disease is rare, but a medical test is fairly accurate. Even if the test has a high true-positive rate, the chance that a positive result actually means disease may be lower than expected because false positives can occur among many healthy people. Bayes’ theorem combines the disease prevalence, the test accuracy, and the test result to compute the probability that the patient truly has the condition.

4.2 Coin-tossing example

Imagine a coin that might be fair or biased. Before any tosses, one may assign probabilities to the possible coin types. After observing a sequence of heads and tails, Bayes’ theorem updates those probabilities based on how likely each sequence would be under each coin model. Repeated observations usually increase confidence in the model that better matches the data.

4.3 Urn and ball example

Consider several urns containing different mixes of colored balls. If one ball is drawn at random and its color is observed, Bayes’ theorem can be used to determine which urn is most likely to have been chosen. The prior probability reflects how likely each urn was before the draw, while the likelihood depends on the color composition of each urn.

4.4 Diagnostic inference example

Bayesian reasoning is also useful in everyday diagnosis, such as identifying the cause of a mechanical failure. If a machine produces a particular sound, the probability of each possible fault can be updated according to how often that sound occurs under each fault condition. The result is a ranked set of explanations rather than a single guess.

5 Applications

Bayes’ theorem is widely used wherever uncertain information must be updated in light of evidence. Its flexibility makes it valuable in both theoretical and practical settings.

5.1 Bayesian inference

In Bayesian inference, Bayes’ theorem is the foundation for learning about unknown parameters and latent variables. Observed data are used to update a prior distribution into a posterior distribution. This approach supports interval estimation, prediction, and uncertainty quantification.

5.2 Statistical decision making

Bayesian decision theory uses posterior probabilities to guide choices under uncertainty. Decisions may be based on expected loss, utility, or risk. This framework is especially useful when outcomes have different costs and when the best action depends on the probability of several competing states.

5.3 Machine learning

Many machine learning methods use Bayesian ideas to estimate model parameters, compare models, or classify new observations. Bayes’ theorem helps formalize how evidence changes the probability of a class label or a latent structure. It is also used in probabilistic graphical models and hierarchical modeling.

5.4 Filtering and prediction

Sequential updating is central to filtering problems, where new measurements arrive over time. Bayes’ theorem allows a system’s state to be revised at each step, producing improved estimates of what is currently true and what may happen next. This principle appears in signal processing, navigation, and time-series analysis.

5.5 Spam detection and classification

In text classification, Bayes’ theorem can help determine whether a message belongs to a particular category, such as spam or not spam. The model compares the probability of the words appearing under each class and updates the class probabilities accordingly. Simple probabilistic classifiers based on this idea are valued for their speed and interpretability.

Several related formulas extend or reframe Bayes’ theorem for different mathematical settings. These variants are common in probability theory, statistics, and applied modeling.

6.1 Bayes’ rule

The term Bayes’ rule is often used interchangeably with Bayes’ theorem. In some contexts, it refers to the algebraic rearrangement that expresses a conditional probability in terms of a reverse conditional probability, a prior, and a normalizing term. The name emphasizes practical use rather than formal derivation.

6.2 Bayes’ theorem for continuous variables

For continuous random variables, probabilities are replaced by probability densities. The posterior density is proportional to the product of the likelihood and the prior density:

\[ p(\theta \mid x) \propto p(x \mid \theta)p(\theta). \]

A normalizing integral is required to make the posterior integrate to one. This version is standard in modern Bayesian statistics.

6.3 Bayes factor

The Bayes factor compares how strongly two hypotheses explain the same evidence. It is defined as the ratio of their marginal likelihoods. When combined with prior odds, it produces posterior odds. Bayes factors are often used in model selection and hypothesis testing.

6.4 Bayes’ formula in odds form

Bayes’ theorem can also be written using odds rather than probabilities. In this form,

\[ \text{Posterior odds} = \text{Prior odds} \times \text{Likelihood ratio}. \]

This expression makes the updating process especially transparent. The evidence modifies the prior odds by a factor showing how much more likely the data are under one hypothesis than another.

7 Assumptions and limitations

Bayes’ theorem is mathematically exact, but its practical use depends on how the prior, likelihood, and model structure are chosen. The quality of the result depends on the assumptions built into the analysis.

7.1 Dependence on prior information

Posterior results can be influenced by the selected prior, particularly when the data set is small or weakly informative. Different priors may lead to noticeably different conclusions in such cases. Careful justification of prior choices is therefore important.

7.2 Sensitivity to model assumptions

The likelihood model must represent the data-generating process reasonably well. If the model omits important features, the posterior may be misleading even when the calculations are correct. Sensitivity analysis is often used to check how robust conclusions are to changes in assumptions.

7.3 Small-sample considerations

With limited data, random variation can dominate the evidence. In these situations, the posterior may remain close to the prior, and conclusions should be interpreted cautiously. Bayesian methods can still be useful in small samples, but uncertainty should be reported clearly.

7.4 Common misinterpretations

A frequent mistake is to confuse \(P(A \mid B)\) with \(P(B \mid A)\). These probabilities are usually different, even though Bayes’ theorem relates them. Another common error is to ignore the base rate, which can lead to exaggerated interpretations of rare events. Careful attention to the full formula helps prevent these misunderstandings.

8 Historical background

Bayes’ theorem emerged from work in the eighteenth century and later became central to statistical reasoning. Its development reflects a gradual shift from isolated probability results to a broad inferential framework.

8.1 Thomas Bayes

Thomas Bayes was an English minister and mathematician associated with early work on inverse probability. His posthumously published essay introduced an argument that can be interpreted as an early form of the theorem now bearing his name. Although the original presentation was limited, it laid important groundwork for later developments.

8.2 Richard Price’s role

Richard Price edited and presented Bayes’ work to the Royal Society. He recognized the significance of the unpublished essay and helped bring it to public attention. His contribution was important in preserving and disseminating the result.

8.3 Later development in Bayesian statistics

In the nineteenth and twentieth centuries, the theorem became a foundation for Bayesian statistics. Researchers extended the basic idea to parameter estimation, decision theory, and probabilistic modeling. Modern computing later made it practical to apply Bayesian methods to large and complex problems.

9 See also

9.1 Conditional probability

A probability that an event occurs given that another event is known to have occurred.

9.2 Law of total probability

A rule for computing the probability of an event by summing over a partition of possible causes.

9.3 Bayes’ theorem in statistics

The use of Bayes’ theorem as a core principle for inference, estimation, and model comparison in statistical analysis.