1 Bayesian framework
1.1 Prior distribution
In Bayesian inference, the prior distribution represents an analyst’s uncertainty about an unknown quantity before observing the current data. It encodes assumptions about plausible parameter values or model structures and can be informed by earlier studies, domain knowledge, or chosen to reflect limited information. The prior is typically expressed as a probability distribution over parameters or hypotheses, allowing the uncertainty to be manipulated mathematically.
1.2 Likelihood function
The likelihood function describes how probable the observed data are under different parameter values. Formally, for parameter \(\theta\) and data \(y\), the likelihood is the function \(p(y\mid \theta)\) viewed as a function of \(\theta\). It captures the statistical model’s mechanism that links parameters to observations and is central to updating uncertainty.
1.3 Bayes’ theorem
Bayes’ theorem provides the rule for updating a prior with data through the likelihood. In its most common form, \[ p(\theta\mid y)=\frac{p(y\mid \theta)p(\theta)}{p(y)}. \] Here \(p(\theta\mid y)\) is the posterior, \(p(y\mid \theta)\) is the likelihood, \(p(\theta)\) is the prior, and \(p(y)\) is the marginal likelihood (also called evidence). Conceptually, the theorem formalizes how evidence reweights prior beliefs according to how well parameter values explain the data.
1.4 Posterior distribution
The posterior distribution is the updated probability distribution of unknown quantities after accounting for observed data. It characterizes uncertainty about parameters, model components, or hypotheses in a way that reflects both prior assumptions and the evidence provided by the data. In Bayesian analysis, the posterior is often the primary target of computation and interpretation.
1.4.1 Posterior density
When the unknown quantity is continuous, the posterior is represented by a probability density function (pdf) \(p(\theta\mid y)\). The density indicates relative plausibility across the parameter space; posterior probabilities for regions are obtained by integrating the density over those regions.
1.4.2 Posterior probability
When the unknown quantity is discrete, the posterior is described by posterior probabilities \(P(\theta\mid y)\) for each possible value of \(\theta\). These probabilities sum to one and provide direct numerical weights for each alternative.
1.4.3 Posterior measure
More generally, Bayesian posteriors can be defined as probability measures rather than densities or point probabilities. This measure-theoretic view is useful when the parameter space has mixed or complicated structure, ensuring that posterior distributions remain well-defined under broad conditions.
2 Mathematical formulation
2.1 Discrete case
For a discrete parameter \(\theta\in\{\theta_1,\dots,\theta_K\}\), Bayes’ theorem becomes \[ P(\theta_k\mid y)=\frac{p(y\mid \theta_k)P(\theta_k)}{\sum_{j=1}^K p(y\mid \theta_j)P(\theta_j)}. \] The denominator ensures normalization. Each posterior probability reflects the prior probability of the parameter value and how strongly the likelihood supports it.
2.2 Continuous case
For a continuous parameter \(\theta\) with prior density \(p(\theta)\) and likelihood \(p(y\mid \theta)\), the posterior density is \[ p(\theta\mid y)=\frac{p(y\mid \theta)p(\theta)}{p(y)}, \quad p(y)=\int p(y\mid \theta)p(\theta)\,d\theta. \] The marginal likelihood is computed by integrating the joint model over parameter values, effectively averaging the likelihood under the prior.
2.3 Posterior proportionality
In many derivations and computations, one uses a proportionality form: \[ p(\theta\mid y)\propto p(y\mid \theta)p(\theta). \] This states that the posterior is proportional to the product of likelihood and prior, with the proportionality constant being the normalizing factor. This expression is especially common when the exact normalizing constant is difficult to obtain but samples or other summaries are still desired.
2.4 Normalizing constant
The normalizing constant \(p(y)\) is given by \[ p(y)=\int p(y\mid \theta)p(\theta)\,d\theta \] (or a corresponding sum in the discrete setting). It ensures the posterior integrates to one. The same quantity also plays a role in model comparison, since different models yield different evidence values.
3 Properties of the posterior distribution
3.1 Conjugacy
Conjugate priors are chosen so that, after multiplying the prior by the likelihood, the posterior falls within the same family as the prior. Conjugacy enables closed-form updates and reduces computational burden. While conjugacy is convenient, it may impose structure that does not always match complex real-world models, so it is often balanced against model fidelity.
3.2 Proper and improper posteriors
A posterior is called proper if it is a valid probability distribution: it has a total mass of one when integrated (or summed) over the parameter space. An improper prior may still lead to a proper posterior, but improper priors require careful treatment because the normalizing constant may diverge. In practice, analysts check that the resulting posterior is normalizable.
3.3 Posterior support
Posterior support refers to the set of parameter values for which the posterior assigns nonzero probability (discrete) or nonzero density (continuous). Support depends on both prior mass and the likelihood’s ability to align with the data. If the prior assigns zero mass to a region, the posterior will also assign zero mass there regardless of how the likelihood behaves in that region.
3.4 Multimodality
A posterior can have multiple modes, producing several distinct regions of high plausibility. Multimodality often arises from non-convex likelihoods, weak identifiability, or mixture models. It complicates numerical computation and interpretation because standard summary measures (such as a single mean) may fail to represent the structure of uncertainty.
4 Common posterior calculations
4.1 Closed-form solutions
In some models, conjugacy or special structure yields closed-form posterior distributions. This allows direct computation of posterior densities, means, and interval summaries. Closed-form results are widely used in introductory Bayesian settings because they provide transparent relationships between prior choices, data, and posterior updates.
4.2 Numerical integration
When closed-form expressions are unavailable, integrals required for normalization and posterior summaries are approximated numerically. Techniques include grid-based methods for low-dimensional parameters and adaptive quadrature or specialized integration strategies for higher dimensions. Numerical integration can be accurate but may become computationally expensive as dimensionality grows.
4.3 Monte Carlo methods
Monte Carlo methods approximate posterior quantities by drawing samples from the posterior or from distributions closely related to it. The central idea is that expectations under the posterior can be estimated from sample averages. These approaches are widely used in Bayesian practice because they scale better than brute-force integration for many problems.
4.3.1 Markov chain Monte Carlo
Markov chain Monte Carlo (MCMC) constructs a Markov chain whose stationary distribution is the posterior. After a burn-in period, samples from the chain are treated as approximate draws from the posterior. Algorithms such as Metropolis–Hastings and Gibbs sampling differ in how proposed moves are generated and accepted.
4.3.2 Importance sampling
Importance sampling estimates posterior expectations by reweighting samples from a proposal distribution. If samples are drawn from \(q(\theta)\) instead of directly from the posterior, weights proportional to \(p(\theta\mid y)/q(\theta)\) adjust for the mismatch. Performance depends on how well the proposal distribution covers the regions where the posterior mass lies.
4.3.3 Sequential Monte Carlo
Sequential Monte Carlo (SMC) methods update an approximation as data arrive or as an artificial sequence of distributions is traversed. Particles carry weights and are resampled to focus computation on relevant regions. SMC is often used in state-space models or when posterior computation is naturally sequential.
4.4 Variational inference
Variational inference approximates the posterior by a simpler distribution chosen from a tractable family. Optimization selects the member of that family that best matches the true posterior according to a divergence measure (commonly the Kullback–Leibler divergence). Variational methods can be substantially faster than MCMC for large datasets, though they may underestimate uncertainty depending on the chosen approximation family.
5 Posterior summaries
5.1 Posterior mean
The posterior mean is the expected value of a parameter under the posterior distribution. It is often used as a point estimate and minimizes squared error loss among unbiased estimators under a Bayesian decision-theoretic viewpoint. The mean may be sensitive to outliers or to heavy-tailed posteriors.
5.2 Posterior median
The posterior median is the value dividing posterior probability mass into two equal parts. It can be more robust than the mean for skewed distributions. Like the mean, it serves as a useful point estimate, especially when distributions are asymmetric.
5.3 Credible intervals
Credible intervals are Bayesian uncertainty intervals derived directly from the posterior distribution. A \((1-\alpha)\) credible interval contains values of a parameter with posterior probability \(1-\alpha\). Unlike frequentist confidence intervals, credible intervals are interpreted as direct statements about posterior probability given the observed data.
5.4 Highest posterior density interval
The highest posterior density (HPD) interval is a credible interval with the smallest possible width among intervals of a given posterior probability content. It is defined so that every point inside the interval has posterior density at least as large as every point outside. HPD intervals can be especially informative for multimodal or skewed posteriors.
5.5 Posterior mode
The posterior mode is the parameter value at which the posterior density (or probability) is maximized. It corresponds to the most likely parameter under the posterior. For multimodal distributions, there may be multiple modes, and the choice between them can affect interpretation.
6 Posterior prediction
6.1 Posterior predictive distribution
Posterior prediction uses the posterior distribution to forecast future or unobserved data. The posterior predictive distribution is \[ p(y_{\text{new}}\mid y)=\int p(y_{\text{new}}\mid \theta)\,p(\theta\mid y)\,d\theta. \] This integrates over parameter uncertainty rather than plugging in a single estimate, typically producing predictive uncertainty that reflects both noise and uncertainty about \(\theta\).
6.2 Predictive checks
Posterior predictive checks assess whether simulated data drawn from the posterior predictive distribution resemble the observed data. By comparing summary statistics or distributional features between observed and replicated datasets, analysts can identify model misfit, such as incorrect variance, tail behavior, or dependence structure.
6.3 Forecasting
Forecasting applies posterior prediction to time-indexed or sequential observations. In Bayesian forecasting, the posterior evolves as new measurements are observed, often using hierarchical or state-space models. Predictive accuracy can be evaluated through scoring rules or calibration diagnostics tied to the posterior predictive distribution.
7 Applications
7.1 Parameter estimation
Bayesian posterior distributions provide a full probability-based description of parameter uncertainty after data are incorporated. Parameter estimation may involve reporting point summaries (mean, median, mode) alongside uncertainty (credible intervals). This framework supports estimation even when parameters are correlated or when uncertainty is non-Gaussian.
7.2 Hypothesis comparison
Bayesian methods compare hypotheses by assessing posterior probabilities or using model-specific summaries. When hypotheses correspond to different parameter settings or model classes, the posterior distribution helps quantify how evidence shifts belief. In discrete settings, posterior probabilities for each hypothesis can be directly computed.
7.3 Model selection
Model selection uses criteria that depend on how well a model predicts under the data and prior. The marginal likelihood (evidence) is a key quantity because it averages the likelihood over the prior and naturally penalizes overly flexible models in many contexts. Bayes factors and related comparisons derive from evidence values.
7.4 Decision theory
Decision theory connects Bayesian inference to actions by specifying a loss function and choosing decisions that minimize expected loss under the posterior. The posterior distribution therefore drives optimal decisions, not only inference about parameters. This perspective clarifies why different posterior summaries can be appropriate under different objectives.
8 Related concepts
8.1 Prior predictive distribution
The prior predictive distribution describes what data might look like before observing the data. It is obtained by averaging the likelihood over the prior: \[ p(y)=\int p(y\mid \theta)p(\theta)\,d\theta. \] This distribution is useful for checking whether the prior induces realistic outcomes and for communicating implications of prior assumptions.
8.2 Maximum a posteriori estimation
Maximum a posteriori (MAP) estimation selects the parameter value that maximizes the posterior. MAP can be viewed as a Bayesian analogue of maximum likelihood, incorporating prior information. It yields a single point estimate, which may be preferable when computation of other summaries is costly or when a mode-based interpretation is desired.
8.3 Marginal likelihood
The marginal likelihood is the evidence \(p(y)\) appearing in Bayes’ theorem. It quantifies how compatible a model is with observed data after averaging over all parameter values weighted by the prior. Marginal likelihoods are central to Bayesian model comparison and can be computed exactly in conjugate cases or approximated numerically otherwise.
8.4 Empirical Bayes methods
Empirical Bayes methods estimate prior hyperparameters from the data and then use them to form an approximate posterior. This approach blends Bayesian modeling with frequentist-style estimation of hyperparameters. It can improve performance when a fully Bayesian treatment is computationally demanding, though it introduces additional dependence on the observed data through the estimated prior.