1 Bayesian framework basics

1.1 Likelihood and posterior

In Bayesian statistics, model parameters (or latent variables) are treated as random quantities. A statistical model specifies how observed data are generated given these unknowns. The likelihood describes the plausibility of the data as a function of the parameter values. The posterior is the updated distribution over unknowns after incorporating the data, typically concentrating more mass near values that better explain the observations.

1.2 Bayes’ theorem and updating

Bayes’ theorem formalizes the update from prior belief to posterior belief. It states that the posterior is proportional to the product of the prior and the likelihood, normalized so the resulting distribution integrates to one. This normalization factor is central in some settings because it accounts for overall model fit to the data.

1.3 Random variables and parameter notation

Bayesian texts use notation that distinguishes random variables (whose values are uncertain before observing data) from parameters and hyperparameters (which govern the distributions). For example, one may denote unknown parameters by \(\theta\), observed data by \(y\), and latent variables by \(z\). Priors are typically written as distributions over \(\theta\), while likelihoods express \(p(y\mid \theta)\). In hierarchical models, hyperparameters control priors at multiple layers.

2 Priors: definition and role

2.1 Prior distribution interpretation

A prior distribution represents beliefs about an unknown quantity before observing current data. These beliefs can arise from previous studies, mechanistic considerations, symmetry assumptions, or deliberate modeling choices. In practice, priors also serve as regularizers: they constrain the range of plausible parameter values and encode expected behavior.

2.2 Prior predictive distribution

The prior predictive distribution describes what data one would expect before observing the actual dataset. It is obtained by integrating over the prior. This construction helps assess whether the modeling assumptions, combined with the prior, generate realistic outcomes. When the prior predictive outcomes appear implausible, the prior may be misaligned with the intended use of the model.

2.3 How priors affect posterior uncertainty

Priors have the strongest impact when data are sparse or noisy. They influence both the center of posterior distributions (through prior mean structure) and their spread (through prior variance and tail behavior). As more observations are added—when the model is identifiable and correctly specified—posterior uncertainty typically shrinks and the influence of the prior diminishes relative to the evidence.

3 Choosing a prior

3.1 Informative vs non-informative priors

Informative priors incorporate substantive knowledge and can materially shape posterior inference, particularly in limited-data regimes. Non-informative priors aim to minimize prior influence, but complete neutrality is rarely achievable because “uninformative” must still be defined relative to a parameterization or scale. As a result, the practical meaning of non-informativeness depends on modeling context.

3.2 Weakly informative priors

Weakly informative priors are designed to gently constrain estimates without overpowering the likelihood. They are commonly used when some regularization is desirable—for example, to prevent extreme parameter values—or when uncertainty should remain broad but not unbounded.

3.3 Elicitation from beliefs or expert knowledge

Priors can be selected by eliciting quantitative beliefs from experts or from subject-matter constraints. This might involve specifying plausible ranges, medians, or tail probabilities for parameters. Care is needed to translate such statements into a coherent distribution, and to ensure the prior reflects the intended uncertainty rather than inadvertently encoding overly sharp assumptions.

3.4 MaxEnt and other principle-based choices

Principle-based methods choose priors that satisfy certain optimality criteria. A prominent approach is maximum entropy (MaxEnt), which selects the distribution with the highest entropy among those consistent with specified moment constraints, thereby avoiding unwarranted additional structure. Other schemes include invariance arguments and Jeffreys-type constructions, each motivated by different desiderata.

3.5 Compatibility with constraints (support, positivity, bounds)

A prior must respect the support and constraints implied by the parameter’s role. For instance, probabilities should lie in \([0,1]\), variances should be nonnegative, and scale parameters often require priors defined on \((0,\infty)\). Choosing a distribution with the wrong support can produce nonsensical posterior mass, sampling failures, or misleading inferences.

4 Common prior families

4.1 Conjugate priors

A prior is conjugate if the posterior belongs to the same family as the prior when combined with a particular likelihood. Conjugacy can yield closed-form posteriors and simplifies computation. Even when conjugacy is not available, the concept helps interpret how priors interact with likelihoods through matching distributional forms.

4.2 Beta priors for proportions

The Beta distribution is commonly used for proportions because it naturally lives on \([0,1]\) and can model a wide range of shapes. In many models for binomial data, the Beta prior leads to a Beta posterior, offering both interpretability and computational convenience.

4.3 Dirichlet priors for categorical probabilities

For categorical probability vectors (multiple outcomes with probabilities summing to one), the Dirichlet distribution provides a standard prior. Its concentration parameters determine how strongly the prior favors particular categories versus a more uniform distribution. With multinomial likelihoods, Dirichlet priors often produce analytically tractable posteriors.

4.4 Normal priors for location parameters

A Normal prior is frequently used for location parameters due to its mathematical convenience and because many real-world uncertainties can be approximated as approximately symmetric around a central value. However, Normal priors may be inappropriate when the parameter is expected to have heavy tails, strict positivity, or known bounds.

4.5 Gamma and inverse-gamma priors for scale/variance

For strictly positive quantities such as rates or scales, Gamma distributions are widely used. Variance-related parameters are sometimes modeled with inverse-gamma priors, particularly in classical conjugate settings. Since variance priors can significantly influence tail behavior, careful tuning is important in weak-data or high-uncertainty contexts.

4.6 Half-normal, half-Cauchy, and other scale priors

When a parameter is constrained to be nonnegative (e.g., standard deviations in random-effects models), half distributions are common. The half-normal and half-Cauchy priors are often chosen to allow substantial mass for larger scales while maintaining the correct support. Different tail heaviness leads to distinct robustness properties.

5 Prior–likelihood interaction

5.1 Posterior concentration with more data

When observations are informative and the model structure is compatible with the data-generating process, the posterior typically becomes more concentrated as sample size grows. Under regularity conditions, the posterior often approaches behavior dominated by the likelihood, shrinking the marginal effect of the prior.

5.2 Sensitivity analysis for prior choice

Because priors can affect conclusions, especially with limited data, a common practice is sensitivity analysis: varying prior specifications and examining how posteriors, predictive distributions, or key summaries change. Large differences indicate that the data alone do not dominate the inference, and that conclusions should be framed with prior dependence in mind.

5.3 Checking implied prior predictive behavior

Before fitting a model, one can simulate from the prior predictive distribution and compare the results to what is plausible in the problem domain. This prior predictive check helps identify priors that are internally inconsistent with the scale of the data or that generate unrealistic outcomes, such as extreme probabilities or implausibly large variances.

5.4 Identifiability and weak data scenarios

In some models, different parameter values can produce nearly indistinguishable likelihoods, leading to weak identifiability. In those cases, priors can remain influential even with moderate sample sizes because the data do not sharply determine the parameters. Diagnostics may include examining posterior correlation structure, effective sample size in MCMC, and whether alternative priors yield stable inferences.

6 Hierarchical (multi-level) priors

6.1 Hyperparameters and partial pooling

Hierarchical priors introduce hyperparameters that govern distributions at lower levels. A key effect is partial pooling: groups with limited observations borrow strength from the overall population through shared hyperparameters, while groups with abundant data can still depart from the population mean. This often improves predictive performance and stabilizes estimates.

6.2 Shrinkage and regularization intuition

Hierarchical modeling commonly yields shrinkage, meaning group-level parameters are pulled toward a global center when data are insufficient to support extreme estimates. This effect acts like regularization but is probabilistically grounded: shrinkage arises from prior structure rather than an external penalty term.

6.3 Empirical Bayes vs fully Bayesian hierarchical models

In empirical Bayes, hyperparameters are estimated from the data (often via marginal likelihood optimization) and then treated as fixed for subsequent inference. In contrast, fully Bayesian hierarchical modeling assigns priors to hyperparameters and infers them jointly with other parameters. Fully Bayesian approaches propagate uncertainty in hyperparameters, while empirical Bayes can be simpler computationally.

6.4 Modeling latent structure with priors

Latent-variable models often rely on hierarchical priors to represent unobserved heterogeneity, such as varying effects across individuals or time. Priors can encode assumptions about smoothness, sparsity, or exchangeability across latent components, thereby shaping both posterior inference and predictive accuracy.

7 Improper and reference priors

7.1 Proper vs improper priors

A proper prior is a distribution that integrates to one. An improper prior does not normalize, often used as a limiting case to express “non-informativeness” within a specific mathematical framework. Improper priors can be useful in deriving posteriors, but they must be handled carefully because they can lead to posteriors that fail to integrate or that depend on arbitrary choices.

7.2 When improper priors can still yield proper posteriors

Even if a prior is improper, the resulting posterior may be proper when the likelihood is sufficiently informative to ensure integrability. Determining propriety depends on both the model and the observed data structure. When propriety holds, the posterior can be interpreted as a valid distribution for inference, but this should not be assumed without verification.

7.3 Reference-prior motivations (conceptual overview)

Reference priors are motivated by the desire to define a prior that yields minimal information in a well-defined sense, often relating to limiting behavior of information measures. They serve as candidates for “default” priors when substantive prior information is absent. Conceptually, the goal is to balance neutrality with mathematical tractability, while acknowledging that prior choice is not entirely arbitrary.

8 Computational aspects

8.1 Posterior computation: analytic vs numeric

When conjugacy or other simplifications apply, posterior distributions can be computed analytically. Otherwise, inference requires numerical methods. Common approaches include sampling-based techniques (such as MCMC) and optimization/approximation methods (such as variational inference), each interacting with prior structure in different ways.

8.2 MCMC with priors

Markov chain Monte Carlo (MCMC) methods generate samples from the posterior using stochastic transitions. Priors influence sampling by affecting posterior geometry—e.g., how strongly parameters are constrained and whether the posterior has multiple modes or heavy tails. Diagnostics often include trace plots, autocorrelation assessment, and checks for convergence and effective sample size.

8.3 Variational inference and prior effects

Variational inference approximates the posterior with a tractable family by solving an optimization problem, often minimizing a divergence measure. Since it returns an approximate distribution, the prior can affect not only the target but also the shape and flexibility of the approximation. In practice, approximation quality may vary depending on how strongly the prior constrains parameter regions.

8.4 Approximate Bayes and moment-matching intuition

Some approximate techniques aim to mimic posterior behavior through moment matching or simplified representations. The prior determines which moments (means, variances, correlations) are emphasized and how uncertainty is represented. These approximations can be fast but may understate uncertainty or misrepresent tail probabilities if the approximation family cannot capture the true posterior.

9 Practical guidance and diagnostics

9.1 Posterior predictive checks

Posterior predictive checks assess whether simulated data from the fitted model resemble the observed dataset. Because simulations depend on the posterior—which depends on the prior—the check indirectly tests whether the combined prior-likelihood model can reproduce salient data features. Discrepancies suggest model misspecification, inadequate prior choices, or both.

9.2 Calibration and coverage concepts

Calibration evaluates whether predictive intervals contain the true outcomes at the expected rates. In Bayesian settings, one can assess whether credible intervals behave like well-calibrated uncertainty measures. Poor calibration can arise from overly concentrated priors, mismatch of prior scale to the likelihood, or systematic model errors.

9.3 Reporting priors transparently

Transparent reporting supports reproducibility and interpretability. Good practice includes stating the prior distributions, including parameterizations and any transformations used, along with justification for their selection. Clear documentation makes it possible to understand how results would change under alternative reasonable priors.

9.4 Common pitfalls (overly strong priors, mismatch of support)

Common errors include choosing priors that are much more concentrated than intended, leading to posterior domination by prior assumptions. Another frequent issue is support mismatch, where the prior places mass in regions incompatible with constraints or with the scale implied by the data. Both problems can produce misleading results, sampling instability, or inconsistent predictive performance.

10.1 Conjugacy

Conjugacy describes the situation where the posterior distribution has the same functional form as the prior after observing data, enabling analytic updates. While useful for computation and intuition, conjugacy should not be mistaken for “correctness”; it is a structural property of the chosen likelihood-prior pair.

10.2 Regularization as Bayesian priors

Many regularization strategies in frequentist learning can be interpreted as Bayesian priors. For example, penalties that discourage large parameter values correspond to prior distributions that assign low probability to those values. This connection helps translate tuning and regularization strength into a probabilistic interpretation.

10.3 Priors in model comparison (model evidence, Bayes factors)

In Bayesian model comparison, prior specification affects model evidence (the marginal likelihood) because it determines how parameter space is weighted. As a result, Bayes factors—which compare evidence between models—can depend on prior choices, especially when models are complex or data are limited. Careful prior design is therefore important for stable comparisons.