The Bayesian method is a statistical approach rooted in Bayes' theorem, which describes the probability of an event based on prior knowledge of conditions that might be related to the event. In contrast to frequentist statistics, Bayesian inference treats unknown parameters as random variables and incorporates prior distributions, updating them with observed data to produce posterior distributions. This framework is widely used in fields such as machine learning, data science, medicine, and engineering for parameter estimation, hypothesis testing, and predictive modeling.
1 Foundations
1.1 Bayes' Theorem
1.1.1 Formal Statement
| Bayes' theorem is expressed mathematically as \( P(A | B) = \frac{P(B | A) \, P(A)}{P(B)} \), where \( A \) and \( B \) are events, \( P(A | B) \) is the conditional probability of \( A \) given \( B \), \( P(B | A) \) is the conditional probability of \( B \) given \( A \), and \( P(A) \) and \( P(B) \) are the marginal probabilities of \( A \) and \( B \) respectively. |
|---|
1.1.2 Derivation from Conditional Probability
| The theorem follows directly from the definition of conditional probability: \( P(A | B) = \frac{P(A \cap B)}{P(B)} \) and \( P(B | A) = \frac{P(A \cap B)}{P(A)} \). Equating the joint probabilities gives \( P(A | B) P(B) = P(B | A) P(A) \), and rearranging yields Bayes' theorem. |
|---|
1.2 Prior Distribution
1.2.1 Informative vs. Non‑informative Priors
Informative priors encode strong prior knowledge about a parameter, often based on previous studies or expert opinion. Non‑informative or diffuse priors, such as uniform distributions or Jeffreys priors, express minimal prior information and are used when no strong prior beliefs exist.
1.2.2 Conjugate Priors
A conjugate prior is a prior distribution that, when combined with a given likelihood, yields a posterior distribution belonging to the same family as the prior. This property simplifies analytical computation. For example, a Beta prior is conjugate for a Bernoulli likelihood, producing a Beta posterior.
1.3 Likelihood Function
| The likelihood function \( \mathcal{L}(\theta | x) = p(x | \theta) \) quantifies the probability of observing the data \( x \) given a specific value of the parameter \( \theta \). It plays a central role in updating prior beliefs: the larger the likelihood for a parameter value, the more that value is supported by the data. |
|---|
1.4 Posterior Distribution
1.4.1 Calculation via Bayes' Rule
| The posterior distribution is obtained by applying Bayes' rule: \( p(\theta | x) = \frac{p(x | \theta) \, p(\theta)}{p(x)} \). It combines the prior distribution and the likelihood, representing the updated belief about the parameter after observing the data. |
|---|
1.4.2 Normalization Constant
| The denominator \( p(x) = \int p(x | \theta) p(\theta) \, d\theta \) is the marginal likelihood or evidence, which ensures that the posterior integrates to one. In many applications, this integral is intractable, motivating computational methods. |
|---|
2 Bayesian Inference
2.1 Parameter Estimation
2.1.1 Maximum a Posteriori (MAP) Estimation
| MAP estimation finds the parameter value that maximizes the posterior distribution: \( \hat{\theta}_{\text{MAP}} = \arg\max_\theta p(\theta | x) \). It is a point estimate that incorporates prior information and can be computed without evaluating the normalization constant. |
|---|
2.1.2 Posterior Mean and Median
| The posterior mean is \( \mathbb{E}[\theta | x] = \int \theta \, p(\theta | x) \, d\theta \), and the posterior median is the value such that the cumulative posterior probability is 0.5. These estimates reflect the center of the posterior distribution and are often preferred when the posterior is asymmetric. |
|---|
2.2 Credible Intervals
2.2.1 Highest Posterior Density (HPD) Interval
An HPD interval is the smallest set of parameter values that contains a specified posterior probability (e.g., 95%). It captures the most probable values and may be discontinuous if the posterior is multimodal.
2.2.2 Equal‑Tail Interval
An equal‑tail interval excludes probability \( \alpha/2 \) from each tail of the posterior distribution. For a 95% interval, the lower bound is the 2.5th percentile and the upper bound the 97.5th percentile. It is straightforward to compute but may include low‑density regions.
2.3 Hypothesis Testing
2.3.1 Bayes Factor
| The Bayes factor compares two competing hypotheses \( H_0 \) and \( H_1 \) as \( BF_{01} = \frac{p(x | H_0)}{p(x | H_1)} \). It quantifies the evidence in the data for one hypothesis over the other, with values greater than 1 favoring \( H_0 \). Bayes factors do not require p‑values and naturally incorporate model complexity. |
|---|
2.3.2 Decision Theory Framework
Bayesian hypothesis testing can be embedded in a decision‑theoretic framework by assigning prior probabilities to hypotheses and specifying a loss function. The optimal decision minimizes the expected posterior loss, providing a principled basis for choosing between hypotheses.
3 Computational Methods
3.1 Markov Chain Monte Carlo (MCMC)
3.1.1 Metropolis‑Hastings Algorithm
| The Metropolis‑Hastings algorithm generates a Markov chain whose stationary distribution is the target posterior. At each step, a candidate value is proposed from a proposal distribution; it is accepted with probability \( \min\left(1, \frac{p(\theta' | x) q(\theta | \theta')}{p(\theta | x) q(\theta' | \theta)}\right) \). This allows sampling from complex posteriors. |
|---|
3.1.2 Gibbs Sampling
Gibbs sampling is an MCMC method that updates each parameter sequentially by sampling from its full conditional distribution, given the current values of all other parameters. It is particularly efficient when conditional distributions are easy to sample from, such as when conjugate priors are used.
3.2 Variational Inference
3.2.1 Mean‑Field Approximation
Mean‑field variational inference approximates the posterior by assuming independence among parameters: \( q(\theta) = \prod_{i} q_i(\theta_i) \). The approximating distributions are chosen to minimize the Kullback–Leibler divergence between \( q \) and the true posterior.
3.2.2 Evidence Lower Bound (ELBO)
The ELBO is a lower bound on the log marginal likelihood: \( \mathcal{L}(q) = \mathbb{E}_q[\log p(x, \theta) - \log q(\theta)] \). Maximizing the ELBO yields the best mean‑field approximation and provides a diagnostic for convergence.
3.3 Approximate Bayesian Computation (ABC)
ABC is used when the likelihood is intractable. It generates samples from the prior, simulates data using a generative model, and accepts parameter sets that produce simulated data sufficiently close to the observed data, based on a distance metric and a tolerance threshold.
4 Extensions and Special Topics
4.1 Hierarchical (Multilevel) Models
4.1.1 Partial Pooling
Hierarchical models allow information to be shared across groups. Partial pooling shrinks group‑level estimates toward a global mean, balancing group‑specific data with overall patterns. This reduces overfitting and improves estimates for groups with few observations.
4.1.2 Exchangeability
Exchangeability is the assumption that the joint distribution of parameters is invariant under permutation of groups. It underlies many hierarchical models, enabling the use of common prior distributions across groups and simplifying inference.
4.2 Nonparametric Bayesian Methods
4.2.1 Dirichlet Process
The Dirichlet process is a prior over probability distributions that allows an infinite number of clusters or components. It is used in mixture models where the number of clusters is not fixed; data points are automatically assigned to an appropriate number of clusters as the sample size grows.
4.2.2 Gaussian Processes
A Gaussian process defines a prior over functions, characterized by a mean function and a covariance kernel. It is used for nonparametric regression and classification, providing uncertainty estimates and flexible modeling of complex patterns.
4.3 Bayesian Model Averaging
Bayesian model averaging accounts for model uncertainty by averaging predictions or parameter estimates across multiple models, weighted by their posterior probabilities. This approach often improves predictive performance and avoids overconfidence in a single model.
5 Applications
5.1 Machine Learning
5.1.1 Bayesian Neural Networks
Bayesian neural networks place priors over network weights, producing a posterior distribution over weights given the data. This yields predictive distributions with uncertainty quantification, regularization, and improved robustness to overfitting.
5.1.2 Probabilistic Graphical Models
Probabilistic graphical models combine graph theory with probability, representing conditional dependencies among random variables. Bayesian inference in these models enables learning structure and making predictions under uncertainty, with applications in classification, filtering, and topic modeling.
5.2 Data Science and Analytics
5.2.1 A/B Testing
Bayesian A/B testing compares two variants by computing the posterior distribution of the conversion rate difference. It provides intuitive summaries, such as the probability that one variant outperforms the other, and can incorporate prior information from previous experiments.
5.2.2 Recommender Systems
Bayesian methods in recommender systems model user preferences and item characteristics using probabilistic matrix factorization. They generate personalized recommendations with uncertainty estimates and adapt to sparse or noisy interaction data.
5.3 Natural Sciences
5.3.1 Phylogenetics
Phylogenetics reconstructs evolutionary relationships from genetic data. Bayesian methods infer phylogenetic trees by incorporating substitution models as likelihoods and prior distributions over tree topologies and branch lengths, producing posterior distributions of trees.
5.3.2 Climate Modeling
Bayesian statistics are applied to climate science for parameter estimation in Earth system models, data assimilation, and projection uncertainty quantification. They combine observational data with model simulations to produce probabilistic forecasts of future climate states.