Bayesian updating - Ciallo Wiki

1 Concept and Mathematical Foundation

1.1 Bayes’ theorem and its components

Bayesian updating is the process of revising a probability model for an unknown quantity after observing data. Bayes’ theorem provides the rule that transforms an initial uncertainty statement into an updated one.

1.1.1 Prior, likelihood, and marginal likelihood

Consider an unknown parameter or latent variable \( \theta \) and observed data \( x \). A Bayesian model specifies:

A prior \(p(\theta)\): the probability distribution representing beliefs before seeing \(x\).
A likelihood \(p(x \mid \theta)\): the plausibility of observing \(x\) given \(\theta\).
The marginal likelihood (also called evidence) \(p(x)\): the total probability of the data under the whole model.

Bayes’ theorem links these quantities: \[ p(\theta \mid x) = \frac{p(x \mid \theta)\,p(\theta)}{p(x)}. \] The evidence term is obtained by averaging the likelihood over the prior: \[ p(x) = \int p(x \mid \theta)\,p(\theta)\,d\theta \] (for continuous \(\theta\); summations replace integrals when \(\theta\) is discrete).

1.1.2 Posterior as an updated belief distribution

The resulting distribution \(p(\theta \mid x)\) is the posterior, representing revised uncertainty after accounting for the observed data. It is “updated” in the sense that it reweights parameter values in proportion to both:

how well those values explain the data (via the likelihood), and
how plausible those values were initially (via the prior).

1.2 Forms of Bayesian updating

Bayesian updating can be expressed at different levels of granularity, depending on whether the unknown quantity is a single value or an entire distribution.

1.2.1 Point-mass (single-parameter) updates

In simple settings, the parameter of interest may effectively be one-dimensional or may be represented by a point estimate with an associated uncertainty. Even when a practitioner reports a single number (such as a mean), the underlying calculation typically updates a distribution over parameters.

A common special case is when the prior and posterior belong to families that admit closed-form updates (conjugate models), producing an updated distribution that is easy to compute.

1.2.2 Distribution-to-distribution updates

More generally, Bayesian updating is distribution-to-distribution: given a prior distribution over \(\theta\), the model produces a posterior distribution over \(\theta\) after observing \(x\). This perspective emphasizes that uncertainty remains probabilistic throughout, rather than being collapsed immediately into a point estimate.

1.3 Coherence and rationality intuition

Bayesian updating is often justified on grounds of coherence: probability assignments should interact with evidence in a way that avoids internal contradictions and supports rational inference.

1.3.1 Probabilistic consistency as a guiding principle

Bayes’ theorem is the rule ensuring that updating is consistent with probability laws. If an agent starts with a prior probability model and adopts a likelihood model for data, then the posterior computed by Bayes’ theorem is the distribution that maintains these assumptions without double-counting or leaving gaps.

1.3.2 Evidence accumulation over repeated observations

When new observations arrive, the Bayesian framework provides a natural mechanism for incorporating them. Under conditional independence (or more general modeling structures), sequential application of Bayes’ theorem yields the same result as updating once with the combined dataset, illustrating how information accumulates over time.

2 Priors and Their Role

2.1 Types of prior distributions

Priors encode modeling assumptions about parameters before data are observed. They can range from highly informative to deliberately weak.

2.1.1 Informative vs. weakly informative priors

Informative priors reflect substantive prior knowledge, such as results from previous studies or domain constraints.
Weakly informative priors aim to stabilize inference without overwhelming the data; they regularize estimates while still allowing substantial movement when observations disagree.

2.1.2 Uninformative and reference priors

Uninformative priors are intended to exert minimal influence, often used as rough starting points when little prior knowledge is available.
Reference priors are constructed to reflect invariance or objective criteria, attempting to formalize “non-informativeness” in a principled way.

Because no prior is literally “no assumptions,” these approaches are best viewed as particular modeling choices.

2.2 Prior sensitivity and robustness

How strongly the posterior depends on prior choices is a key practical question.

2.2.1 Diagnosing when priors matter most

Prior sensitivity is typically most pronounced when data are limited, the likelihood is relatively flat over plausible parameter regions, or the model is complex (making it easier for prior structure to guide inference). With abundant data, the likelihood often dominates, shrinking the influence of the prior.

2.2.2 Strategies for assessing sensitivity

Common ways to evaluate robustness include:

comparing posteriors under different plausible priors,
examining posterior changes in key quantities (means, credible intervals, predictive distributions),
and using prior predictive checks to ensure the model can generate data patterns consistent with experience.

2.3 Conjugate priors

Conjugacy is a property that makes Bayesian updating computationally convenient.

2.3.1 Conjugacy as computational convenience

A prior is conjugate to a likelihood family if the posterior distribution belongs to the same family as the prior after observing data. This yields straightforward update formulas and avoids repeated numerical integration.

2.3.2 Common conjugate pairs

In many textbook examples, conjugate pairs appear as:

Beta priors with Binomial likelihoods (Beta-Binomial),
Normal priors with Normal likelihoods when variance is known (Normal-Normal),
and related structured combinations for exponential-family models.

Conjugacy is not required for Bayesian updating, but it is a frequent source of analytic solutions.

3 Likelihood and Evidence

3.1 Modeling assumptions in the likelihood

Bayesian updating depends on the likelihood’s representation of how data are generated from parameters.

3.1.1 Choosing a likelihood family

Selecting a likelihood family involves specifying:

the distributional form of observation noise,
the relationship between parameters and expected outcomes,
and sometimes covariate effects.

A good choice aligns the statistical assumptions with observed behavior (e.g., heavy tails, skewness, bounded outcomes).

3.1.2 Independence and factorization

When observations are conditionally independent given parameters, the likelihood factors into a product of terms. This factorization often simplifies both computation and interpretation, and it supports sequential updating via straightforward multiplication of likelihood contributions.

3.2 Evidence interpretation

The marginal likelihood has interpretive and computational roles.

3.2.1 Marginal likelihood (model evidence)

The evidence term \(p(x)\) normalizes the posterior and can also be used for model comparison. It reflects not only how well a model fits the data, but also how much parameter space supports that fit.

3.2.2 Updating under model mismatch

If the likelihood model is misspecified—e.g., incorrect noise assumptions—Bayesian updating still produces a mathematically valid posterior for the assumed model. However, that posterior may not correspond to the true data-generating process, potentially leading to misleading uncertainty estimates.

3.3 Data quality and measurement noise

Observed data are often noisy, incomplete, or biased relative to what the model intends to measure.

3.3.1 Handling uncertainty in observations

Bayesian models can incorporate observation error directly through the likelihood. By explicitly modeling noise, the posterior can propagate measurement uncertainty into parameter uncertainty and predictive intervals.

3.3.2 Robust likelihood choices

Robustness can be improved by choosing likelihoods that better match empirical features—such as using heavy-tailed distributions for outliers—or by introducing latent variables that account for systematic deviations.

4 Practical Bayesian Updating

4.1 Sequential (online) updating

Sequential updating incorporates data as they arrive, updating the posterior repeatedly.

4.1.1 Updating with new batches of data

If a first dataset produces a posterior \(p(\theta \mid x_{1})\), then observing a second batch \(x_{2}\) can be handled by treating the first posterior as the new prior: \[ p(\theta \mid x_{1}, x_{2}) \propto p(x_{2}\mid \theta)\,p(\theta \mid x_{1}). \] This uses the same Bayesian principle with the prior/posterior roles shifted.

4.1.2 Handling streaming data

In streaming contexts, the model updates after each new observation or small chunk. Practical implementations may store sufficient statistics (when available) or use approximations to keep computation feasible.

4.2 Batch updating vs. sequential updating

Whether updating once with all data or multiple times yields the same result depends on assumptions and computation.

4.2.1 When they coincide

Under exact arithmetic and consistent likelihood factorization, sequential and batch updates coincide: multiplying likelihood terms in different groupings leads to the same posterior.

4.2.2 When numerical differences arise

In practice, floating-point arithmetic, approximations, and resampling methods can introduce small numerical discrepancies. These differences often reflect implementation details rather than conceptual disagreement.

4.3 Computational methods

Computing the posterior is sometimes straightforward and sometimes challenging, especially for complex models.

4.3.1 Analytic updates with conjugate models

When conjugacy applies, posterior distributions may have closed-form expressions. This supports fast updates and clear uncertainty quantification.

4.3.2 Approximate inference (e.g., Laplace)

Approximate methods replace an intractable posterior with a simpler approximation. A Laplace approximation, for instance, approximates the posterior near its maximum a posteriori point with a Gaussian distribution, yielding efficient estimates of uncertainty.

4.3.3 Simulation-based inference (e.g., MCMC)

When direct computation is impractical, Markov chain Monte Carlo methods generate samples from the posterior. These samples allow estimation of posterior expectations, tail probabilities, and predictive quantities, subject to convergence diagnostics and sufficient computational effort.

4.4 Parameter estimation and posterior summaries

Even though Bayesian inference yields distributions, summaries are often required for communication and decisions.

4.4.1 Posterior mean, median, and mode

Common point summaries include:

Posterior mean: expectation under the posterior; sensitive to tail behavior.
Posterior median: central tendency robust to asymmetry.
Posterior mode: most probable parameter value (MAP), useful when peaks are well-defined.

Each summary captures a different aspect of the posterior.

4.4.2 Credible intervals and uncertainty quantification

A credible interval is an interval with a posterior probability mass (e.g., 95%) rather than a frequentist coverage guarantee. Credible intervals quantify uncertainty about parameters under the model, and they align naturally with posterior sampling outputs.

4.5 Decision-making using the posterior

Bayesian updating can support decisions via predictive reasoning and loss-based evaluation.

4.5.1 Posterior predictive reasoning

The posterior predictive distribution describes what data one would expect next, averaging over uncertainty in parameters: \[ p(x_{\text{new}} \mid x) = \int p(x_{\text{new}} \mid \theta)\,p(\theta \mid x)\,d\theta. \] This approach enables prediction while respecting parameter uncertainty.

4.5.2 Expected loss and Bayes risk (overview)

Given an action and a loss function, Bayesian decision theory selects actions that minimize expected loss under the posterior. The overall minimal value is sometimes referred to as Bayes risk, tying inference to utility.

5 Examples and Worked Illustrations

5.1 Bernoulli trials and Beta-Binomial updating

Suppose observations are Bernoulli outcomes (success/failure) with success probability \(\theta\). A Beta prior \( \text{Beta}(\alpha,\beta) \) is conjugate to the Binomial likelihood for counts. After observing \(s\) successes out of \(n\) trials, the posterior becomes: \[ \theta \mid \text{data} \sim \text{Beta}(\alpha+s,\ \beta+n-s). \] This simple update shows how prior pseudo-counts add to observed counts.

5.2 Gaussian mean with known variance (Normal-Normal)

Let data be normally distributed with known variance \(\sigma^2\) and unknown mean \(\mu\): \[ x_i \sim \mathcal{N}(\mu,\sigma^2). \] With a Normal prior \(\mu \sim \mathcal{N}(\mu_0,\tau_0^2)\), the posterior is also Normal. The update combines prior information and sample information, typically weighting them by their relative variances: more precise data (smaller \(\sigma^2\) or larger \(n\)) increases the influence of the likelihood.

5.3 Gaussian with unknown mean and variance (outline)

When both mean and variance are unknown, standard conjugate setups often use Normal-Inverse-Gamma or related families. The posterior updates both location (mean) and scale (variance) using observed dispersion. While the algebra can be more involved than in the known-variance case, the conceptual pattern remains: the posterior reflects evidence in both central tendency and variability.

5.4 Posterior predictive checks (light overview)

Posterior predictive checks assess whether simulated future data from the posterior resemble observed data. This is not a proof of model correctness, but it is a practical diagnostic.

5.4.1 Comparing predictions to observed data

A typical workflow samples parameters from the posterior, generates replicated datasets from the likelihood, and compares summary statistics (such as means, variances, or quantiles) between replicated and observed data. Large, systematic mismatches suggest the model may be inadequate.

6 Epistemological Implications

6.1 Bayesian updating as a theory of belief revision

Bayesian updating is frequently presented as a formal framework for revising beliefs in light of evidence.

6.1.1 How evidence updates credences

In this view, probabilities represent degrees of belief. Observations alter those degrees through the likelihood function, producing posterior beliefs that incorporate both new information and prior expectations.

6.2 Learning and convergence (intuition)

As more data accumulate, the posterior often concentrates around parameter values that best explain the observed evidence, though details depend on the model and assumptions.

6.2.1 Increasing data and posterior concentration

With enough information, the likelihood typically overwhelms prior differences, sharpening the posterior distribution. This concentration is reflected in shrinking credible intervals and increasingly stable posterior predictive behavior.

6.3 Limitations and common pitfalls (non-controversial overview)

Bayesian updating is not immune to modeling errors or practical missteps.

6.3.1 Mis-specified models and misleading posteriors

If the assumed likelihood and prior structure do not match the data-generating process, the posterior may give a confident but incorrect picture. Bayesian validity is conditional on the model assumptions.

6.3.2 Overconfidence from poor assumptions

Inadequate treatment of noise, overly narrow priors, or unjustified independence can produce posteriors with unrealistically small uncertainty. Diagnostics such as posterior predictive checks can help reveal such problems.

6.4 Relationship to alternatives (high-level)

Bayesian methods are one way to formalize inference and uncertainty.

6.4.1 Frequentist vs. Bayesian perspectives (conceptual)

Frequentist approaches often emphasize long-run behavior of estimators under repeated sampling, while Bayesian approaches emphasize probability statements about parameters given observed data. Both yield practical tools, but they differ in interpretation and how uncertainty is quantified.

7 Connections to Broader Methods

7.1 Bayesian inference in machine learning

Bayesian updating is a core ingredient in many machine learning frameworks that aim to quantify uncertainty.

7.1.1 Bayesian model selection (overview)

Model comparison can be performed using evidence (marginal likelihood). Competing models are weighed according to how well they explain data while accounting for complexity implicitly through integration over parameters.

7.2 Hierarchical Bayesian updating

Hierarchical models introduce multiple layers of parameters, allowing partial pooling across related groups.

7.2.1 Hyperpriors and partial pooling (conceptual)

In hierarchical Bayes, group-level parameters are themselves random variables governed by higher-level distributions. Hyperpriors express beliefs about these higher-level parameters. The resulting partial pooling balances group-specific variation with overall regularities.

7.3 Bayesian filtering

Bayesian filtering provides a sequential updating framework for systems that evolve over time.

7.3.1 State estimation over time (conceptual linkage)

Filtering methods maintain a distribution over latent states as new observations arrive. This is conceptually aligned with Bayesian updating, with time dynamics incorporated into the likelihood/prior structure.

7.4 Probabilistic programming (overview)

Probabilistic programming systems express Bayesian models and automatically support posterior inference workflows.

7.4.1 Expressing updating workflows in software

In such tools, users define the generative model, and the system handles inference (exact, approximate, or simulation-based). This lowers the barrier to applying Bayesian updating to complex models and data.

8 Notation, Conventions, and Glossary

8.1 Common symbols and definitions

Typical notation includes:

\(\theta\): unknown parameter(s) or latent variables
\(x\): observed data
\(p(\theta)\): prior distribution
\(p(x \mid \theta)\): likelihood
\(p(\theta \mid x)\): posterior distribution
\(p(x)\): marginal likelihood or evidence

8.2 Key terms (prior, likelihood, posterior, evidence)

Prior: distribution encoding uncertainty before observing the data.
Likelihood: function describing how likely the data are under each parameter value.
Posterior: distribution after incorporating the observed evidence.
Evidence (marginal likelihood): normalization constant and basis for model comparison.

8.3 Frequent conceptual misunderstandings (brief)

Common misunderstandings include confusing priors with “probability of the data,” treating the posterior as mere re-labeling without new information, or assuming that Bayesian updating guarantees correctness despite model misspecification. Clarifying the conditioning structure—what is random, given what—is central to correct interpretation.