1 Foundations of Bayesian statistics
Bayesian statistics is a framework for reasoning under uncertainty. It treats probability as a measure of belief or plausibility rather than as a long-run frequency alone. In this view, statistical analysis begins with prior knowledge, incorporates new observations, and produces updated conclusions that can be revised again when more evidence appears.
1.1 Probability as uncertainty
In Bayesian statistics, probability represents uncertainty about unknown quantities such as parameters, future observations, or hypotheses. This interpretation makes it possible to assign probabilistic statements to states of knowledge, not only to repeatable random events. The approach is especially useful when information is incomplete and decisions must be made before certainty is possible.
1.2 Bayes’ theorem
Bayes’ theorem is the mathematical rule that connects prior beliefs, data, and updated beliefs. It states that the posterior probability is proportional to the product of the prior probability and the likelihood of the observed data. The theorem provides the central mechanism by which Bayesian analysis revises expectations in light of evidence.
1.3 Prior, likelihood, and posterior
Bayesian inference is built from three main components: the prior, the likelihood, and the posterior. The prior summarizes what is known before observing data. The likelihood measures how well different parameter values explain the observed data. The posterior combines both and expresses the updated state of knowledge after the data are taken into account.
1.3.1 Prior distributions
Prior distributions encode assumptions, background knowledge, or expert judgment about unknown quantities. They may be informative, weakly informative, or intentionally broad, depending on the problem. A prior can reflect previous studies, physical constraints, or practical expectations, and it helps stabilize inference when data are limited.
1.3.2 Likelihood functions
A likelihood function describes the probability of the observed data as a function of the model parameters. It is determined by the chosen statistical model and the assumed data-generating process. In Bayesian analysis, the likelihood plays a central role in indicating which parameter values are most compatible with the evidence.
1.3.3 Posterior distributions
The posterior distribution is the result of combining the prior and the likelihood. It gives the updated probability of unknown quantities after observing the data. Rather than yielding a single estimate alone, it describes a full range of plausible values and allows uncertainty to be carried through later calculations.
1.4 Bayesian inference
Bayesian inference uses the posterior distribution to estimate parameters, compare hypotheses, and predict new observations. Summary measures such as posterior means, medians, modes, and intervals are often reported. Because the posterior contains a complete probabilistic description, it supports both estimation and prediction within one coherent framework.
2 Bayesian models
Bayesian models apply the general framework of prior and posterior reasoning to specific statistical problems. They range from simple one-parameter models to flexible structures for complex data. Model choice depends on the data type, the scientific question, and the degree of uncertainty that must be represented.
2.1 Conjugate priors
Conjugate priors are priors chosen so that the posterior distribution belongs to the same family as the prior. This property simplifies calculation and sometimes allows closed-form solutions. They are useful in introductory analysis and in settings where rapid updating is needed, though they can be less flexible than modern alternatives.
2.2 Hierarchical models
Hierarchical models describe data with multiple levels of variation, such as observations nested within groups. They allow parameters to be shared across related units, producing partial pooling and more stable estimates. Such models are especially valuable when groups are small or unevenly sampled, because information can be borrowed across the hierarchy.
2.3 Bayesian regression
Bayesian regression extends regression analysis by placing distributions on coefficients and other unknown quantities. The resulting posterior captures both effect size and uncertainty. This approach is used for linear, nonlinear, and generalized regression settings, and it is particularly helpful when predictors are numerous or when regularization is desired.
2.4 Bayesian classification
Bayesian classification assigns class probabilities using Bayes’ theorem and a probabilistic model for features within each class. Instead of producing only a label, it provides uncertainty over possible categories. This makes it useful in decision systems, pattern recognition, and predictive modeling where confidence matters as much as the final assignment.
2.5 Bayesian nonparametrics
Bayesian nonparametrics refers to models whose complexity can grow with the data rather than being fixed in advance. These methods can represent flexible shapes, clusters, or latent structures without specifying a single finite-dimensional form. They are often used when the true complexity of the phenomenon is unknown or difficult to constrain beforehand.
3 Computational methods
Many Bayesian models cannot be solved exactly, especially when the posterior distribution has a complicated form. Computational methods therefore play a major role in modern practice. These methods approximate posterior quantities, predictive distributions, and model evidence with manageable numerical procedures.
3.1 Analytical solutions
Analytical solutions are available in certain simple models, often when conjugate priors are used. In such cases, the posterior can be derived in closed form and summarized directly. Although limited in scope, these solutions are valuable for theory, teaching, and as benchmarks for more advanced algorithms.
3.2 Markov chain Monte Carlo
Markov chain Monte Carlo methods generate samples from a posterior distribution by constructing a stochastic process whose long-run behavior matches the target distribution. The samples can then be used to approximate means, intervals, and other quantities of interest. These methods are widely used because they can handle highly complex posterior shapes.
3.2.1 Metropolis–Hastings algorithm
The Metropolis–Hastings algorithm proposes candidate values and accepts or rejects them according to a rule that preserves the target posterior distribution. It is flexible and conceptually straightforward, making it one of the foundational Monte Carlo methods in Bayesian computation. Its performance depends on the choice of proposal mechanism and tuning parameters.
3.2.2 Gibbs sampling
Gibbs sampling updates one parameter or group of parameters at a time from its conditional distribution given the others. This strategy can simplify simulation when full conditionals are easy to sample. It is especially effective in models with structured dependence, though mixing may be slow in difficult problems.
3.2.3 Hamiltonian Monte Carlo
Hamiltonian Monte Carlo uses gradient information to move efficiently through the posterior distribution. By following paths inspired by physical dynamics, it can explore high-dimensional spaces more effectively than many basic random-walk methods. It is now a standard tool in advanced Bayesian computation because of its speed and accuracy in many applications.
3.3 Variational inference
Variational inference replaces sampling with optimization. It approximates the posterior by a simpler distribution chosen from a tractable family, then adjusts that approximation to be close to the true posterior. This approach is often faster than Monte Carlo methods, making it attractive for large datasets and complex machine learning models.
3.4 Approximate Bayesian computation
Approximate Bayesian computation is used when the likelihood is difficult or impossible to evaluate directly but data can be simulated from the model. It compares simulated and observed data using summary statistics and accepts parameter values that produce close matches. The method is especially useful in fields with complex generative models.
4 Model evaluation and comparison
Bayesian analysis includes tools for checking whether a model fits the data and for comparing competing models. Good practice requires assessing not only how well a model explains observed outcomes, but also how reliably it generalizes to new cases. Evaluation methods help identify weaknesses and guide model refinement.
4.1 Posterior predictive checking
Posterior predictive checking compares observed data with data simulated from the posterior predictive distribution. If the model is adequate, replicated data should resemble the observed data in relevant features. Discrepancies can reveal missing structure, poor assumptions, or unusual observations that deserve closer examination.
4.2 Bayes factors
Bayes factors compare the relative support that the data provide for two models or hypotheses. They are based on the ratio of marginal likelihoods and express evidence in probabilistic terms. Bayes factors can be informative, though their interpretation depends on the models being compared and the priors assigned to them.
4.3 Marginal likelihood
The marginal likelihood is the probability of the observed data under a model after averaging over parameter uncertainty. It plays a central role in model comparison and Bayesian evidence calculations. Because it integrates over the parameter space, it rewards models that fit well while accounting for model complexity.
4.4 Information criteria
Information criteria provide approximate measures for comparing models using fit and complexity penalties. In Bayesian work, they are often used when exact evidence calculations are difficult. These criteria offer practical guidance, although they may not capture every aspect of posterior uncertainty or predictive performance.
5 Decision-making under uncertainty
Bayesian methods support decisions by linking probability with consequences. Rather than focusing only on estimation, this perspective asks how different actions perform under uncertainty. The result is a principled way to choose among options when outcomes are not known in advance.
5.1 Bayesian decision theory
Bayesian decision theory selects actions by minimizing expected loss under the posterior distribution. It formalizes the idea that a good decision depends on both uncertainty and the costs of possible outcomes. This framework is widely used in medicine, engineering, economics, and automated systems.
5.2 Loss functions
A loss function assigns a numerical cost to each possible decision and outcome. Different loss functions lead to different optimal choices, such as posterior means, medians, or modes. The selection of a loss function reflects the goals of the analysis and the consequences of error.
5.3 Credible intervals
Credible intervals are Bayesian uncertainty intervals that contain the parameter with a stated posterior probability. For example, a 95% credible interval indicates a 95% probability that the parameter lies within the interval, given the model and data. This interpretation differs from frequentist confidence intervals and is often easier to communicate.
5.4 Prediction intervals
Prediction intervals describe the range in which future observations are expected to fall with a specified probability. They combine parameter uncertainty with the variability of new data. In Bayesian analysis, prediction intervals are derived from the posterior predictive distribution and are useful for planning and forecasting.
6 Applications
Bayesian methods appear in many disciplines because they provide a coherent way to incorporate prior knowledge and uncertainty. Their flexibility makes them suitable for both simple estimation problems and highly structured modern models. As computational tools have improved, their practical use has expanded substantially.
6.1 Scientific research
In scientific research, Bayesian methods are used to estimate parameters, evaluate hypotheses, and combine evidence from multiple studies. They are valuable when sample sizes are small, measurements are noisy, or prior findings matter. The framework also supports sequential learning, where conclusions are updated as new experiments are completed.
6.2 Machine learning
Bayesian ideas are central to many machine learning methods, including probabilistic modeling, regularization, and uncertainty quantification. They help prevent overfitting by incorporating prior structure and by averaging over plausible models. Bayesian approaches are especially useful when predictions must include confidence estimates rather than point predictions alone.
6.3 Medicine and public health
In medicine and public health, Bayesian analysis supports diagnosis, treatment evaluation, and surveillance. It can combine clinical knowledge with observed patient data, making it useful in situations where evidence is incomplete or arrives over time. The framework is also helpful for interpreting diagnostic tests and monitoring population-level trends.
6.4 Engineering and quality control
Engineering applications include reliability analysis, fault detection, sensor fusion, and process monitoring. Bayesian methods can integrate expert judgment with measurements from complex systems, improving decision-making under uncertainty. In quality control, they help estimate defect rates and assess whether production processes are behaving as expected.
6.5 Economics and social sciences
Economics and social science research often involves uncertain relationships, heterogeneous groups, and limited data. Bayesian models can represent latent factors, time variation, and differences across populations. They are used in forecasting, survey analysis, behavioral modeling, and the study of decision processes.
7 History and development
Bayesian statistics developed over several centuries through contributions from mathematicians, astronomers, and statisticians. Its historical trajectory includes early theoretical ideas, classical formalization, and a major revival driven by computation. The modern field combines these strands into a widely used statistical paradigm.
7.1 Thomas Bayes and early foundations
Thomas Bayes is associated with the foundational theorem that bears his name. His work, developed in the eighteenth century, laid the basis for updating probabilities from observed evidence. Early Bayesian ideas were initially limited in influence but established an enduring principle of inference under uncertainty.
7.2 Laplace and classical Bayesian theory
Pierre-Simon Laplace expanded Bayesian reasoning into a broad analytical framework. He applied probabilistic methods to astronomy, natural science, and decision problems, and he helped formalize the use of prior information and posterior updating. His work made Bayesian ideas more systematic and influential in classical statistics.
7.3 Modern Bayesian revival
In the twentieth century, Bayesian statistics regained prominence as statisticians recognized its coherence and flexibility. The revival was supported by developments in decision theory, subjective probability, and applied modeling. This period also saw clearer distinctions between Bayesian and frequentist approaches, stimulating debate and further refinement.
7.4 Computational revolution
The growth of computing transformed Bayesian statistics from a largely theoretical framework into a practical methodology for complex problems. Simulation algorithms, numerical optimization, and efficient software made it possible to analyze models that were previously intractable. This change greatly expanded the range of realistic Bayesian applications.
8 Criticism and limitations
Despite its strengths, Bayesian statistics has limitations that affect interpretation, implementation, and acceptance. Some concerns arise from modeling choices, while others involve computational demands or philosophical differences about probability. Careful practice is needed to avoid overconfidence and misuse.
8.1 Choice of priors
The choice of prior can influence conclusions, especially when data are sparse. While priors can incorporate useful knowledge, they can also introduce subjectivity or produce results that depend strongly on assumptions. For this reason, prior specification is often examined carefully and justified through domain knowledge or robustness checks.
8.2 Computational complexity
Many Bayesian models require substantial computation, especially in high-dimensional or highly nonlinear settings. Sampling and approximation methods may be slow, difficult to tune, or sensitive to numerical issues. This complexity can limit accessibility and make large-scale applications challenging without specialized tools.
8.3 Sensitivity analysis
Sensitivity analysis studies how conclusions change when priors, model assumptions, or computational settings are altered. It is an important safeguard because it reveals whether results are stable or overly dependent on a particular specification. In Bayesian work, such checks help distinguish strong evidence from fragile inference.
8.4 Subjectivity and interpretability
Some critics argue that Bayesian analysis gives too much weight to subjective choices, especially in the selection of priors and models. Others note that Bayesian output can be difficult to interpret for users unfamiliar with probabilistic reasoning. Supporters respond that these choices are explicit rather than hidden, and that the framework often improves transparency by making assumptions visible.