1 Statement of Bayes’ Rule

1.1 Conditional probability recap

Bayes’ rule builds directly on conditional probability, which quantifies the chance of one event occurring given that another event is known to have occurred. If \(A\) and \(B\) are events with \(P(B)>0\), then the conditional probability of \(A\) given \(B\) is defined by \[ P(A\mid B)=\frac{P(A\cap B)}{P(B)}. \] This relationship lets one express joint probabilities in terms of conditionals and vice versa.

1.2 The core formula

Bayes’ rule reverses the conditioning direction. For events \(A\) and \(B\) with \(P(B)>0\), \[ P(A\mid B)=\frac{P(B\mid A)\,P(A)}{P(B)}. \] Here, \(P(A)\) is the prior belief about \(A\), \(P(B\mid A)\) describes how likely the evidence \(B\) is if \(A\) were true, and \(P(B)\) ensures the result sums to 1 across the relevant possibilities for \(A\).

1.3 Equivalent forms and rearrangements

Bayes’ rule can be rearranged to highlight different components. Using the identity \(P(B)=P(B\mid A)P(A)+P(B\mid \neg A)P(\neg A)\) for a binary \(A\), the denominator becomes the total probability of observing \(B\) under all competing explanations. In general settings, Bayes’ rule is also written in terms of random variables: \[ P(H\mid E)=\frac{P(E\mid H)\,P(H)}{P(E)}, \] where \(H\) denotes a hypothesis and \(E\) denotes observed evidence. For discrete cases, \(P(E)\) can be expanded as a sum over hypotheses; for continuous cases, it is expressed using probability densities.

2 Interpreting Priors, Likelihoods, and Posteriors

2.1 Priors (initial belief)

A prior \(P(A)\) represents the probability assigned to an event or hypothesis before observing the new evidence \(B\). Priors may come from historical data, domain knowledge, or deliberately chosen weakly informative assumptions. Their role is to encode uncertainty in a way that can be revised rather than treated as fixed truth.

2.2 Likelihood (evidence given a hypothesis)

The likelihood term \(P(B\mid A)\) measures how well the hypothesis \(A\) predicts the observed evidence \(B\). Higher likelihood means the evidence is more consistent with \(A\). In practice, likelihood is often computed from an assumed statistical model that links hidden causes (hypotheses or parameters) to observable data.

2.3 Posterior (updated belief)

The posterior \(P(A\mid B)\) is the revised probability of the hypothesis after incorporating the evidence. Conceptually, it weights the prior by how strongly the evidence supports that hypothesis relative to alternatives. The posterior is the quantity used for subsequent reasoning, predictions, or decisions.

2.4 Normalization and why the denominator matters

The denominator \(P(B)\) prevents the posterior from exceeding valid probability bounds. It acts as a normalization factor: \[ P(B)=\sum_{i} P(B\mid A_i)P(A_i) \] for mutually exclusive hypotheses \(\{A_i\}\) in discrete settings, or an analogous integral in continuous settings. Without this term, comparisons across hypotheses would be uncalibrated: the same numerator structure could yield different total masses depending on how common the evidence is overall.

3 Worked Examples

3.1 Diagnostic testing (base rate example)

Consider a population where a disease \(D\) affects a fraction \(P(D)=0.01\). A test returns positive \(T^+\). Suppose the test has sensitivity \(P(T^+\mid D)=0.99\) and a false-positive rate \(P(T^+\mid \neg D)=0.05\). Bayes’ rule gives \[ P(D\mid T^+)=\frac{P(T^+\mid D)P(D)}{P(T^+\mid D)P(D)+P(T^+\mid \neg D)P(\neg D)}. \] Substituting values: \[ P(D\mid T^+)=\frac{0.99\cdot 0.01}{0.99\cdot 0.01+0.05\cdot 0.99}\approx \frac{0.0099}{0.0099+0.0495}\approx 0.167. \] Even with strong sensitivity, the relatively low base rate \(P(D)\) can keep the posterior probability moderate.

3.2 Coin and dice problems

A common classroom scenario uses a coin with two possible types: fair \(F\) and biased \(B\). Let \(P(F)=0.5\) and \(P(B)=0.5\). Suppose \(P(H\mid F)=0.5\) for heads and \(P(H\mid B)=0.75\). After observing one head \(H\), \[ P(F\mid H)=\frac{P(H\mid F)P(F)}{P(H\mid F)P(F)+P(H\mid B)P(B)} =\frac{0.5\cdot 0.5}{0.5\cdot 0.5+0.75\cdot 0.5} =\frac{0.25}{0.625}=0.4. \] A similar logic applies to dice: if two dice differ in the probability of rolling a particular face, posterior beliefs shift based on observed outcomes and the relative prior plausibility of each die.

3.3 Bayesian reasoning with multiple outcomes

When evidence can take multiple values, Bayes’ rule distributes posterior mass across hypotheses accordingly. For hypotheses \(\{A_i\}\) and observed evidence \(E=e\), \[ P(A_i\mid E=e)=\frac{P(E=e\mid A_i)P(A_i)}{\sum_j P(E=e\mid A_j)P(A_j)}. \] This formulation ensures that if the likelihood function strongly favors one hypothesis for the observed value, that hypothesis receives a larger share of posterior probability, while others shrink proportionally.

4 Algebraic and Computational Aspects

4.1 Computing the evidence term (marginal likelihood)

In practice, the challenging quantity is often the evidence \(P(B)\), also called the marginal likelihood or total probability of the evidence. For discrete hypotheses \(\{A_i\}\), \[ P(B)=\sum_i P(B\mid A_i)P(A_i). \] For continuous latent variables, the sum becomes an integral over the hypothesis space. Computing \(P(B)\) can be straightforward in low-dimensional, discrete problems, but becomes costly when the hypothesis space is large or when models involve complex dependencies.

4.2 Handling small probabilities and numerical stability

Direct multiplication of probabilities can underflow in numerical computing when values are very small. A standard remedy is to operate in log space: compute log-likelihoods and log-priors, then normalize. For example, one may use \[ \log P(A_i\mid B)=\log P(B\mid A_i)+\log P(A_i)-\log P(B), \] where \(\log P(B)\) is obtained via a stable log-sum-exp computation. This approach preserves accuracy and avoids floating-point underflow.

4.3 Common simplifications and shortcuts

Several algebraic shortcuts are frequently used:

  • Binary hypothesis shortcut: If only \(A\) vs. \(\neg A\) matters, the denominator can be written as two terms, reflecting both possibilities.
  • Proportionality: Often one can write

\[ P(A\mid B)\propto P(B\mid A)P(A), \] compute unnormalized scores for each hypothesis, and then normalize across them.

  • Symmetry arguments: If hypotheses are symmetric under the observation process, priors and likelihoods may coincide for subsets, reducing the number of distinct computations.

5.1 Naive Bayes and conditional independence

Naive Bayes applies Bayes’ rule to classification problems by assuming conditional independence among features given the class label. If features are \(X_1,\dots,X_n\) and class is \(C\), then \[ P(C\mid X_1,\dots,X_n)\propto P(C)\prod_{k=1}^n P(X_k\mid C). \] Although the independence assumption is often unrealistic, the method can still perform well because it captures how evidence combines and often estimates probability ratios effectively.

5.2 Bayes estimators and Bayesian updating

Beyond computing \(P(A\mid B)\), Bayesian methods use posteriors to estimate unknown quantities. A Bayes estimator is a function of the posterior distribution chosen to minimize expected loss under that posterior. For example, under squared-error loss, the posterior mean is optimal; under absolute-error loss, the posterior median is optimal. Bayesian updating refers to how a posterior distribution is updated as new data arrives.

5.3 Bayes factors (evidence comparison)

Bayes factors compare hypotheses by quantifying how strongly evidence favors one over another: \[ \text{BF}_{12}=\frac{P(E\mid H_1)}{P(E\mid H_2)}. \] Unlike posterior probabilities, Bayes factors separate the evidence contribution from prior weighting. Posterior odds can be obtained by multiplying the Bayes factor by the prior odds.

5.4 Connection to conditional probability trees

Conditional probability trees visualize Bayes’ rule by representing successive branching probabilities: one can start from priors and multiply by conditional likelihoods to compute joint probabilities, then regroup those joints to obtain the reverse conditional. The “tree” perspective clarifies that Bayes’ rule is essentially a consistent rearrangement of joint probability mass across different orderings of conditional events.

6 Bayesian Updating Over Time

6.1 Sequential evidence and repeated updates

When evidence arrives in stages, Bayesian updating can be applied repeatedly. If \(E_1\) is observed first and \(E_2\) later, then \[ P(H\mid E_1,E_2)=\frac{P(E_2\mid H,E_1)P(H\mid E_1)}{P(E_2\mid E_1)}. \] Under certain modeling assumptions where \(E_2\) is conditionally independent of \(E_1\) given \(H\), this simplifies to using \(P(E_2\mid H)\). Sequential application typically produces the same result as updating once with the combined evidence, provided the probabilistic model is coherent.

6.2 Conjugate priors (conceptual overview)

A prior is conjugate to a likelihood family if the posterior remains in the same parametric family after observing data. This property yields closed-form updates and simplifies computation. Conjugacy is widely used for tractable Bayesian inference—for instance, when modeling Bernoulli trials with beta priors, the posterior is also beta. The main conceptual benefit is interpretability and computational efficiency, though conjugate pairs may not always be the best fit to real-world structure.

6.3 Posterior predictive reasoning

Posterior predictive distributions describe what future observations are expected after accounting for current data. If \(Y\) is a future random variable and \(H\) is the hypothesis or parameter, then \[ P(Y\mid \text{data})=\int P(Y\mid H)\,P(H\mid \text{data})\,dH \] in continuous settings (or a sum in discrete ones). This approach naturally incorporates uncertainty in the unknown quantity by averaging predictions over the posterior.

7 Assumptions, Limitations, and Common Pitfalls

7.1 Misinterpreting likelihood vs. posterior

A frequent mistake is to interpret likelihood \(P(B\mid A)\) as if it were the posterior probability that \(A\) is true after seeing \(B\). Likelihood is a support score for how plausible the evidence is under a hypothesis; it does not automatically translate into belief without the prior and normalization. In effect, likelihood is “evidence-for” a hypothesis, whereas posterior is “hypothesis-after” the evidence.

7.2 Ignoring base rates

Base rates \(P(A)\) strongly influence posteriors, particularly when events are rare. Even highly sensitive tests may yield limited posterior probability if false positives and the overall prevalence make the evidence common among non-hypothesis cases. This is not an error in Bayes’ rule, but a reminder that prior probabilities capture crucial context.

7.3 Dependence between variables

Bayes’ rule itself remains valid without independence assumptions; problems arise when additional modeling choices are made incorrectly. For example, in naive Bayes, conditional independence is assumed. If features are strongly correlated given the class, the model’s factorization can misestimate probabilities. Similarly, dependence between evidence variables can make sequential updating formulas invalid unless the dependency structure is modeled accurately.

8 Bayes’ Rule in Practice

8.1 From probability theory to statistical modeling

In applied work, Bayes’ rule connects mathematical probability to statistical models. A model specifies:

  1. a hypothesis space or parameterization,
  2. a prior distribution over that space,
  3. a likelihood that links parameters to observations.

Once these pieces are chosen, Bayes’ rule produces the posterior distribution used for inference.

8.2 Parameter estimation vs. event inference

Bayesian inference can target either:

  • Event inference: computing probabilities of discrete hypotheses (e.g., which model generated the data).
  • Parameter estimation: learning values of continuous parameters (e.g., regression coefficients).

Both use the same core mechanism: update beliefs by combining priors with likelihood information and normalizing.

8.3 Decision-making with expected outcomes

Bayesian reasoning often culminates in decisions based on expected performance under uncertainty. Using the posterior predictive distribution, one can compute expected loss or expected utility for each available action and choose the action that optimizes the criterion. This framing makes probability outputs directly actionable rather than purely descriptive.

9 Quick Reference

9.1 Notation cheat sheet

  • \(A\), \(H\): hypothesis or event of interest
  • \(B\), \(E\): observed evidence
  • \(P(A)\): prior probability of \(A\)
  • \(P(B\mid A)\): likelihood of observing \(B\) given \(A\)
  • \(P(A\mid B)\): posterior probability after observing \(B\)
  • \(P(B)\): evidence (marginal likelihood) used for normalization

9.2 Typical workflow for applying Bayes’ rule

  1. Define hypotheses and evidence, and choose a prior \(P(A)\).
  2. Specify the likelihood \(P(B\mid A)\) from a statistical model.
  3. Compute the denominator \(P(B)\) via total probability (sum or integral).
  4. Obtain the posterior \(P(A\mid B)\).
  5. Use the posterior for prediction, estimation, or decision-making.

9.3 Common example templates

  • Binary template: compute

\[ P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B\mid A)P(A)+P(B\mid \neg A)P(\neg A)}. \]

  • Multi-hypothesis template: compute unnormalized weights \(w_i=P(B\mid A_i)P(A_i)\), then normalize:

\[ P(A_i\mid B)=\frac{w_i}{\sum_j w_j}. \]

  • Sequential template: update stepwise using the posterior from the previous stage as the new prior.