Prior probability, in Bayesian statistics, represents the initial degree of belief in an event or hypothesis before new evidence is taken into account. It is a fundamental component of Bayes' theorem, allowing the combination of prior knowledge with observed data to produce a posterior probability. The choice of prior can be subjective or objective, influencing the inference process.
1 Definition and Basic Concept
A prior probability is the probability distribution that expresses one's uncertainty about a parameter before data are observed. It formalizes existing knowledge, expert opinion, or a neutral starting point into a mathematical form. In Bayesian inference, the prior is combined with the likelihood of observed data to yield the posterior distribution, which updates the initial belief.
1.1 Distinction from Posterior Probability
The posterior probability is the revised belief after accounting for new evidence, calculated by Bayes' theorem. While the prior represents knowledge *before* data, the posterior represents knowledge *after* data. As more data accumulate, the posterior increasingly dominates the prior, diminishing its influence.
1.2 Role in Bayes' Theorem
Bayes' theorem provides the mathematical rule for updating prior beliefs. The prior appears as the denominator's scaling factor, ensuring that the posterior is a valid probability distribution.
1.2.1 Mathematical Formulation
For a parameter *θ* and data *x*, Bayes' theorem is:
\[
| P(\theta | x) = \frac{P(x | \theta) \, P(\theta)}{P(x)} |
|---|
\]
| where *P*(θ) is the prior probability, *P*(x | θ) is the likelihood, *P*(x) is the marginal likelihood (evidence), and *P*(θ | x) is the posterior probability. |
|---|
1.3 Relationship to Frequentist Probability
In frequentist statistics, parameters are fixed unknown constants, and probabilities refer to long-run frequencies of events. There is no concept of a prior distribution for a parameter. Bayesian statistics treats parameters as random variables, using priors to quantify uncertainty. The two philosophies differ in interpretation: frequentists rely solely on the likelihood, while Bayesians incorporate prior information.
2 Types of Prior Distributions
Priors can be classified by how much information they incorporate and whether they are proper (integrating to one).
2.1 Uninformative (Non-informative) Priors
Uninformative priors attempt to exert minimal influence on the posterior, letting the data speak for themselves. They aim to represent ignorance about the parameter.
2.1.1 Uniform Priors
A uniform prior assigns equal probability to all possible values of a parameter over a specified range. For example, a uniform prior over [0,1] for a probability parameter. Uniform priors are not invariant under reparameterization—a uniform prior on a parameter may not be uniform on a transformed version.
2.1.2 Jeffreys Priors
Jeffreys priors are designed to be invariant under reparameterization. For a parameter θ, the Jeffreys prior is proportional to the square root of the Fisher information: *P*(θ) ∝ √*I*(θ). These priors are often uninformative in the sense of being flat on the natural scale of the parameter's information geometry.
2.2 Informative Priors
Informative priors incorporate substantial prior knowledge, such as from previous studies or expert opinion. They can sharpen inference when data are limited.
2.2.1 Conjugate Priors
A conjugate prior, when combined with a given likelihood, yields a posterior distribution belonging to the same family as the prior. This mathematical convenience simplifies computation, especially before the widespread use of Markov chain Monte Carlo.
2.2.1.1 Example: Beta-Binomial Conjugate Pair
For a binomial likelihood with success probability *p*, the conjugate prior is a beta distribution: *p* ~ Beta(α, β). After observing *k* successes in *n* trials, the posterior is Beta(α + *k*, β + *n* – *k*). The hyperparameters α and β can be interpreted as prior "successes" and "failures."
2.3 Improper Priors
An improper prior is a probability distribution that does not integrate to a finite value over its domain. It is often used to represent a lack of prior information (e.g., a uniform prior over an unbounded range). Improper priors can be mathematically convenient but must be used with caution.
2.3.1 Conditions for Proper Posterior
An improper prior is acceptable only if the resulting posterior is proper (i.e., integrates to one). This requires that the likelihood provides sufficient information to "fill in" the missing mass. For example, the improper uniform prior over ( –∞, ∞) for a normal mean yields a proper posterior when at least one observation is available.
2.4 Weakly Informative Priors
Weakly informative priors provide some structure (e.g., a reasonable scale) without strongly influencing the posterior. They lie between uninformative and informative priors. For instance, a Cauchy(0, 2.5) prior on a logistic regression coefficient is weakly informative: it is proper and regularizes extreme values but does not impose strong prior beliefs.
3 Selecting and Eliciting Priors
Choosing a prior requires balancing objectivity, available knowledge, and computational tractability.
3.1 Subjective Prior Elicitation
Subjective prior elicitation involves formal methods to convert expert knowledge into a probability distribution. Techniques include probability encoding (asking experts for quantiles or matching bets), the use of histograms, and interactive software. The goal is to capture genuine uncertainty without overconfidence.
3.2 Empirical Priors
Empirical priors are derived from existing data, often from earlier studies or related populations. For example, a meta-analysis of previous trials can inform the prior for a new clinical study. This approach can be considered semi-Bayesian because the prior itself is estimated from data.
3.3 Hierarchical Prior Modeling
In hierarchical Bayesian models, priors themselves have hyperparameters with their own priors (hyperpriors). This approach allows borrowing strength across groups. For example, in a multi-center clinical trial, each center's treatment effect may have a prior centered on an overall mean, with the variance of that prior estimated from data.
4 Influence of Priors on Inference
The impact of a prior on posterior inference depends on the amount and informativeness of the data.
4.1 Prior Sensitivity Analysis
A prior sensitivity analysis assesses how changes in the prior affect posterior conclusions. Common methods include comparing results under different priors (e.g., informative vs. vague) or using global and local sensitivity measures. Robust conclusions are those that do not change dramatically under plausible prior variations.
4.2 Asymptotic Dominance by Likelihood
As the sample size grows, the likelihood overwhelms the prior, and the posterior converges to the same result regardless of the prior (provided the prior does not assign zero mass to the true parameter value). This property, known as the Bernstein–von Mises theorem, ensures that Bayesian inference is asymptotically objective.
4.3 Controversies and Debates in Prior Choice
The choice of prior has sparked philosophical and practical debates, especially regarding subjectivity and the replicability of analyses.
4.3.1 Subjectivity vs. Objectivity
Subjective Bayesians argue that all inference is inherently personal and that prior choice should explicitly reflect an individual's beliefs. Objective Bayesians advocate for priors that are minimally informative or agreed upon by a community, aiming for inferences that are "reference" or "default." This tension continues in methodological discussions, though many practitioners adopt pragmatic approaches that balance both views.
5 Applications
Bayesian methods with explicit priors are widely used across scientific and engineering fields.
5.1 Machine Learning and Bayesian Neural Networks
In machine learning, priors regularize models. For neural networks, a prior over weights (e.g., Gaussian) corresponds to L2 regularization. Bayesian neural networks use priors to capture uncertainty in predictions, enabling probabilistic outputs and better calibration.
5.2 Clinical Trials and Adaptive Designs
Clinical trials often employ Bayesian methods. Priors can incorporate historical control data, reducing the required sample size. In adaptive designs, interim data update the posterior, allowing modifications like early stopping or dose adjustment. Regulatory agencies accept Bayesian approaches when priors are carefully justified.
5.3 Economics and Forecasting
Economists use Bayesian methods to combine prior beliefs from economic theory with observed data. For example, in dynamic stochastic general equilibrium (DSGE) models, priors help identify parameters that are otherwise weakly identifiable. Bayesian forecasting produces predictive distributions that account for parameter uncertainty.