A Bayesian network is a probabilistic graphical model that represents a set of variables and their conditional dependencies via a directed acyclic graph (DAG). Each node corresponds to a random variable, and edges denote direct probabilistic influences, quantified by conditional probability tables. The network encodes the joint probability distribution over all variables, enabling efficient inference and learning from data. Bayesian networks are widely used in artificial intelligence, machine learning, statistics, and various applied fields for reasoning under uncertainty.

1 Fundamental concepts

1.1 Graph theory representation

1.1.1 Directed acyclic graph (DAG)

A directed acyclic graph is a finite graph with directed edges and no directed cycles. In the context of a Bayesian network, the DAG encodes conditional independence assumptions. The absence of cycles ensures that the graph defines a consistent factorization of the joint distribution.

1.1.2 Nodes and edges

Nodes represent random variables, which may be discrete or continuous. Edges represent direct probabilistic dependencies. If there is an edge from node A to node B, then A is a parent of B, and B is conditionally dependent on A given its other parents.

1.2 Conditional probability tables (CPTs)

For each node, a conditional probability table specifies the probability of the node taking each possible value given every combination of values of its parents. In discrete Bayesian networks, these tables store the parameters of the local conditional distributions.

1.3 Chain rule and factorization

The joint probability distribution over all variables is factorized as the product of the conditional probabilities of each node given its parents, following the chain rule of probability applied to the DAG. For a network with variables \(X_1, X_2, \ldots, X_n\), the joint distribution is \(P(X_1, X_2, \ldots, X_n) = \prod_{i=1}^n P(X_i \mid \text{Parents}(X_i))\).

1.4 Markov property and d‑separation

1.4.1 Conditional independence

The Markov property states that each node is conditionally independent of its non-descendants given its parents. This property is used to derive conditional independence relations among variables. d‑separation (directed separation) is a graphical criterion to determine whether a set of nodes is conditionally independent of another set given a third set; it is based on blocking paths in the DAG.

1.4.2 Moral graphs and ancestral graphs

A moral graph is obtained by adding undirected edges between all pairs of parents of a common child (marrying the parents) and then dropping the directions of all edges. Moral graphs are used in inference algorithms such as the junction tree algorithm. Ancestral graphs generalize DAGs to represent conditional independence structures that may arise from latent variables and selection bias.

2 Types of inference

2.1 Exact inference

Exact inference computes the posterior probability of a set of query variables given evidence, exploiting the factorization of the joint distribution.

2.1.1 Variable elimination

Variable elimination is an algorithm that sums out (eliminates) one variable at a time by summing over its values after multiplying relevant factors. The order of elimination affects computational efficiency.

2.1.2 Junction tree algorithm

The junction tree algorithm (also called the clique tree or tree of cliques) first moralizes the DAG, triangulates it, and constructs a tree of cliques. It then performs message passing (belief propagation) on the tree to compute marginals for all variables exactly.

2.2 Approximate inference

When exact inference is computationally intractable (e.g., for large or multiply connected networks), approximate methods are used.

2.2.1 Sampling methods

Sampling methods generate random samples from the distribution to estimate posterior probabilities.

2.2.1.1 Rejection sampling

Rejection sampling generates samples from the prior distribution and accepts only those that are consistent with the observed evidence. It can be inefficient when the evidence is unlikely.

2.2.1.2 Importance sampling

Importance sampling uses a proposal distribution to generate samples, weighting each by the ratio of the target distribution to the proposal distribution. Likelihood weighting is a variant that generates samples by fixing evidence and weighting accordingly.

2.2.1.3 Markov chain Monte Carlo (MCMC)

MCMC methods, such as Gibbs sampling, construct a Markov chain whose stationary distribution is the posterior. Samples are drawn sequentially, and after burn-in, they approximate the target distribution.

2.2.2 Variational inference

Variational inference approximates the posterior by optimizing a simpler distribution (e.g., mean-field family) to minimize the Kullback–Leibler divergence. It is often faster than sampling for large models.

2.3 Causal inference

Bayesian networks can be interpreted as causal models when edges represent causal relationships. Causal inference goes beyond probabilistic conditioning to answer questions about interventions and counterfactuals.

2.3.1 Interventions and do‑calculus

An intervention, denoted by do(X = x), sets a variable X to a value x, removing its dependence on parents. The do-calculus provides rules to transform expressions involving interventions into standard conditional probabilities that can be estimated from observational data.

2.3.2 Counterfactual reasoning

Counterfactuals consider what would have happened if an event had been different, given observed evidence. Pearl's structural causal model framework treats counterfactuals as computed by modifying the causal model and using the observed state.

3 Learning Bayesian networks

3.1 Parameter learning

Parameter learning estimates the conditional probability tables given a fixed network structure and a dataset of observations.

3.1.1 Maximum likelihood estimation

Maximum likelihood estimation (MLE) computes the parameters that maximize the likelihood of the data. For discrete networks, the MLE is the empirical frequency of each parent-child configuration.

3.1.2 Bayesian estimation and conjugate priors

Bayesian estimation incorporates prior knowledge by placing a prior distribution (e.g., Dirichlet) over the parameters. The posterior is the product of the prior and the likelihood, and the prior is conjugate if the posterior has the same functional form.

3.2 Structure learning

Structure learning infers the DAG itself from data. It is a challenging combinatorial search problem.

3.2.1 Score‑based methods

Score-based methods assign a score to each candidate structure and search for the structure with the best score (e.g., using greedy hill climbing or simulated annealing).

3.2.1.1 Bayesian information criterion (BIC)

BIC is a penalized likelihood score: \(\text{BIC} = \log P(D \mid \hat{\theta}, G) - \frac{d}{2} \log N\), where \(d\) is the number of free parameters and \(N\) the sample size. It trades off fit and model complexity.

3.2.1.2 Bayesian Dirichlet equivalence (BDe)

BDe is a Bayesian score based on the marginal likelihood assuming Dirichlet priors. It satisfies score equivalence: all DAGs in the same Markov equivalence class receive the same score.

3.2.2 Constraint‑based methods

Constraint-based methods use conditional independence tests to infer edges, then orient them using d‑separation rules.

3.2.2.1 PC algorithm

The PC algorithm (named after Peter Spirtes and Clark Glymour) starts with a complete undirected graph and removes edges based on conditional independence tests. It then orients edges using a set of rules to produce a completed partially directed acyclic graph (CPDAG).

3.2.2.2 Fast causal inference (FCI)

FCI extends the PC algorithm to handle latent confounders. It identifies a partial graph (PAG) that represents causal relationships under the presence of unmeasured variables.

3.2.3 Hybrid methods

Hybrid methods combine score-based and constraint-based approaches. For example, they use conditional independence tests to restrict the search space and then apply a score-based search within the reduced space.

4 Applications

4.1 Diagnosis and medical reasoning

Bayesian networks model the probabilistic relationships between diseases and symptoms. They support differential diagnosis, test selection, and prognosis. Examples include the Pathfinder system for lymph node pathology and the Help system for ventilator management.

4.2 Reliability engineering and fault detection

In reliability engineering, Bayesian networks model component failures and their dependencies. They are used for fault tree analysis, system reliability assessment, and root cause diagnosis in complex systems such as aircraft or power grids.

4.3 Natural language processing

Bayesian networks appear in semantic parsing, document classification, and language modeling. For instance, they can be used for word sense disambiguation by modeling latent topics and contextual dependencies.

4.4 Bioinformatics and gene regulatory networks

Bayesian networks are applied to infer gene regulatory networks from expression data, model protein‑protein interactions, and analyze genetic pathways. Their ability to handle uncertainty and integrate multiple data sources makes them valuable in systems biology.

4.5 Financial risk assessment

In finance, Bayesian networks model dependencies among market variables, credit risk factors, and portfolio returns. They are used for stress testing, credit scoring, and operational risk analysis.

5.1 Dynamic Bayesian networks

Dynamic Bayesian networks (DBNs) extend Bayesian networks to model temporal processes. They replicate the network structure across time slices and add transition probabilities between slices. DBNs are used for speech recognition, robot localization, and time‑series modeling (e.g., hidden Markov models are a special case).

5.2 Influence diagrams (decision networks)

Influence diagrams extend Bayesian networks by adding decision nodes (actions) and utility nodes (preferences). They model decision‑making under uncertainty and are used for optimal policy selection in artificial intelligence and operations research.

5.3 Object‑oriented Bayesian networks

Object‑oriented Bayesian networks (OOBNs) organize networks hierarchically using classes and instances. Reusable modules (classes) encapsulate subgraphs, facilitating model construction and maintenance for large, complex domains.

5.4 Bayesian network classifiers

Bayesian network classifiers are restricted structures designed for classification tasks.

5.4.1 Naive Bayes

The naive Bayes classifier assumes that all features are conditionally independent given the class variable. It corresponds to a Bayesian network with the class as the root and features as children. Despite its strong independence assumption, it performs well in many applications.

5.4.2 Tree‑augmented Naive Bayes (TAN)

TAN relaxes the independence assumption by allowing each feature to have at most one additional parent from among the other features, forming a tree structure among features. This captures some dependencies while retaining computational tractability.