1 Definition and Support

1.1 Simplex and probability-vector constraints

The Dirichlet distribution is defined over the probability simplex: the set of vectors \(x=(x_1,\dots,x_K)\) such that \(x_i\ge 0\) for all \(i\) and \(\sum_{i=1}^{K} x_i=1\). A random vector with a Dirichlet distribution can therefore be interpreted as an uncertain probability vector, where each component represents the proportion assigned to one category.

1.2 Parameterization (concentration parameters)

For \(K\) categories, the distribution is parameterized by a vector of positive concentration parameters \(\boldsymbol{\alpha}=(\alpha_1,\dots,\alpha_K)\) with \(\alpha_i>0\). Larger values of \(\alpha_i\) typically make the corresponding component less variable, while smaller values increase variability and can concentrate mass near the simplex boundaries.

A common derived quantity is the total concentration \[ \alpha_0=\sum_{i=1}^{K}\alpha_i, \] which governs overall “strength” of the prior across all categories.

1.3 Probability density function (PDF)

Let \(x\) lie on the simplex. The Dirichlet distribution with parameters \(\boldsymbol{\alpha}\) has density \[ p(x\mid \boldsymbol{\alpha})=\frac{1}{B(\boldsymbol{\alpha})}\prod_{i=1}^{K} x_i^{\alpha_i-1}, \] where \(B(\boldsymbol{\alpha})\) is the multivariate Beta function, serving as a normalization constant.

1.4 Special cases and relationship to the Beta distribution

When \(K=2\), the simplex reduces to pairs \((x_1,x_2)\) with \(x_2=1-x_1\). In this case, the Dirichlet distribution becomes the Beta distribution: \[ (x_1, x_2)\sim \mathrm{Dir}(\alpha_1,\alpha_2)\quad \Longleftrightarrow \quad x_1\sim \mathrm{Beta}(\alpha_1,\alpha_2). \] This provides a direct multi-dimensional generalization of the Beta family.

2 Construction and Intuition

2.1 Gamma-variable representation

A standard construction generates a Dirichlet random vector by normalizing independent Gamma variables. Draw \[ Y_i \sim \mathrm{Gamma}(\alpha_i,1)\quad \text{independently,} \] and set \[ X_i=\frac{Y_i}{\sum_{j=1}^{K}Y_j}. \] The resulting vector \(X=(X_1,\dots,X_K)\) lies on the simplex and follows \(\mathrm{Dir}(\boldsymbol{\alpha})\). This representation is widely used for sampling and for deriving moments.

2.2 Stick-breaking viewpoint

Another intuition imagines distributing a “stick” of length 1 among categories sequentially. In a stick-breaking construction, one assigns a fraction to the first component and then recursively breaks the remaining part among the remaining components. While multiple stick-breaking variants exist, they all reflect the idea that a Dirichlet sample can be built from dependent sequential proportions whose joint distribution matches the Dirichlet density.

2.3 Interpretation of concentration/strength

The parameters \(\alpha_i\) control both central tendency and dispersion. Roughly, when \(\alpha_0\) is large, the distribution concentrates around its mean vector \(\mathbb{E}[X_i]=\alpha_i/\alpha_0\). When \(\alpha_0\) is small, the distribution becomes more diffuse and can allocate substantial probability mass near edges of the simplex, allowing some components to be near zero in typical draws.

2.4 Symmetric Dirichlet and uniformity

If all parameters are equal, \(\alpha_i=\alpha\), the distribution is symmetric under permutations of categories. The mean is uniform: \[ \mathbb{E}[X_i]=\frac{1}{K}. \] For \(\alpha=1\), the density is constant on the simplex, yielding a uniform distribution over probability vectors. Values \(\alpha<1\) tend to produce sparse vectors (often with some near-zero components), whereas \(\alpha>1\) yields more evenly spread proportions.

3 Core Properties

3.1 Normalization constant (multivariate Beta function)

The normalization constant is \[ B(\boldsymbol{\alpha})=\frac{\prod_{i=1}^{K}\Gamma(\alpha_i)}{\Gamma(\alpha_0)}. \] With this choice, the density integrates to 1 over the simplex. The multivariate Beta function generalizes the familiar Beta normalizer used in the two-component case.

3.2 Moments

3.2.1 Mean of each component

For \(X\sim \mathrm{Dir}(\boldsymbol{\alpha})\), \[ \mathbb{E}[X_i]=\frac{\alpha_i}{\alpha_0}. \] Thus the mean weights are proportional to the concentration parameters.

3.2.2 Variance and covariance structure

Variances and covariances reflect the constrained nature of the simplex (components must sum to one). Let \(i\neq j\). Then \[ \mathrm{Var}(X_i)=\frac{\alpha_i(\alpha_0-\alpha_i)}{\alpha_0^2(\alpha_0+1)}, \] and \[ \mathrm{Cov}(X_i,X_j)=-\frac{\alpha_i\alpha_j}{\alpha_0^2(\alpha_0+1)}. \] The negative covariance indicates that increasing one component typically decreases others under the sum constraint.

3.2.3 Higher-order moments

Higher moments can be expressed using rising factorials (Pochhammer symbols) or Gamma-function ratios. In general, Dirichlet moments can be computed in closed form for products of components, which makes the distribution analytically convenient for probabilistic modeling and for checking inference methods.

3.3 Marginal and conditional distributions

3.3.1 Marginal Beta distributions

Any single component \(X_i\) has a Beta distribution when the other components are marginalized out. Specifically, \[ X_i \sim \mathrm{Beta}\left(\alpha_i,\alpha_0-\alpha_i\right). \] This provides a tractable one-dimensional view of uncertainty for each category probability.

3.3.2 Aggregation (summing categories)

Sums of subsets of Dirichlet components are also distributed according to Beta laws. More generally, if you group categories into two blocks, the total probability assigned to one block follows a Beta distribution whose parameters are sums of the relevant \(\alpha_i\). This property supports flexible modeling when categories are combined or coarse-grained.

3.3.3 Conditional distributions (Bayesian updating views)

Conditioning on parts of the simplex yields distributions that remain Dirichlet in reduced dimension after appropriate normalization. This stability under conditioning is closely related to conjugacy results in Bayesian inference, where prior updates preserve the Dirichlet form.

4 Conjugacy in Bayesian Models

4.1 Multinomial likelihood

Consider categorical outcomes with \(K\) categories and probabilities \(p=(p_1,\dots,p_K)\) on the simplex. A multinomial experiment produces counts \(n=(n_1,\dots,n_K)\) with likelihood \[ p(n\mid p)=\frac{N!}{\prod_{i=1}^{K} n_i!}\prod_{i=1}^{K} p_i^{n_i}, \] where \(N=\sum_i n_i\). Here, the unknown probability vector \(p\) is modeled with a Dirichlet prior.

4.2 Posterior update rules (Dirichlet-multinomial)

If the prior is \(p\sim \mathrm{Dir}(\boldsymbol{\alpha})\), then the posterior after observing counts \(n\) is \[ p\mid n \sim \mathrm{Dir}(\alpha_1+n_1,\dots,\alpha_K+n_K). \] This “additive” update is a hallmark of Dirichlet-multinomial conjugacy, greatly simplifying Bayesian computation.

4.3 Predictive distributions (Dirichlet-multinomial / Polya’s urn connection)

Integrating out \(p\) yields the predictive distribution for future categorical observations. The resulting predictive probabilities follow a Dirichlet-multinomial form, which is mathematically equivalent to the behavior of Polya’s urn. In that analogy, each observed draw increases the effective counts associated with the category drawn, reinforcing its future selection probability.

4.4 Example: updating with observed counts

Suppose \(K=3\) and a prior \(\boldsymbol{\alpha}=(2,2,2)\). After observing counts \(n=(5,1,0)\), the posterior becomes \[ \boldsymbol{\alpha}'=(2+5,\;2+1,\;2+0)=(7,3,2). \] The posterior mean probabilities are \(\alpha'_i/\alpha'_0\), providing updated expected category proportions while preserving a full distribution over uncertainty.

5 Sampling and Computation

5.1 Sampling via normalized Gamma draws

Sampling is typically performed using the Gamma representation. One samples \(Y_i\sim \mathrm{Gamma}(\alpha_i,1)\) independently, then sets \(X_i=Y_i/\sum_j Y_j\). This method is straightforward, general for any \(K\), and aligns naturally with common scientific computing libraries.

5.2 Sampling stability and numerical considerations

Practical implementations must handle cases where \(\alpha_i\) is very small or very large. When parameters are near zero, sampled Gamma variates can underflow or concentrate heavily near zero; when parameters are large, computations of normalization and log-densities can suffer from precision loss. Using log-space computations and robust normalization strategies improves stability.

5.3 Estimation of parameters

5.3.1 Method of moments

Given samples of \(X\), one can estimate \(\boldsymbol{\alpha}\) from empirical means and variances. Because the Dirichlet mean is \(\alpha_i/\alpha_0\) and the variance depends on \(\alpha_0\), moment-matching can recover both relative weights and total concentration. This approach is simple but may be sensitive to sample size and to the presence of extreme sparsity.

5.3.2 Maximum likelihood and digamma-based methods

Maximum likelihood estimation generally requires solving nonlinear equations involving the digamma function \(\psi(\cdot)\), since the log-likelihood under a Dirichlet includes terms like \(\log \Gamma(\alpha_i)\) and \( (\alpha_i-1)\log x_i\). Iterative procedures (e.g., Newton or fixed-point updates) are commonly used, often requiring careful initialization and convergence checks.

5.3.3 MAP estimation with priors on parameters

In settings where uncertainty about \(\boldsymbol{\alpha}\) itself is relevant, one may place hyperpriors and compute a maximum a posteriori estimate. MAP estimation leads to additional regularization, which can help prevent degenerate solutions, particularly when data are sparse or when some categories have few or zero observations.

6 Applications

6.1 Modeling uncertainty over categorical probabilities

The Dirichlet distribution is widely used to represent uncertainty over a probability vector in categorical models. It provides a principled way to encode prior beliefs (through \(\boldsymbol{\alpha}\)) and to update those beliefs with observed category counts in a closed form.

6.2 Topic modeling and Latent Dirichlet Allocation (LDA)

In LDA, documents are represented as mixtures of latent topics, and topics generate words according to categorical distributions. The mixture weights for topics per document are commonly modeled with a Dirichlet prior, enabling Bayesian inference of topic proportions and supporting probabilistic interpretation of uncertainty.

6.3 Compositional data analysis

Compositional data are vectors of nonnegative components summing to one (for example, relative abundances). The Dirichlet distribution offers a baseline probabilistic model for such data, yielding coherent marginals and a natural covariance structure compatible with the simplex constraint. It is often used either directly or as a starting point for more flexible alternatives.

6.4 Random mixture weights in generative models

Many generative approaches require random mixture weights over components (e.g., mixture models). Drawing weights from a Dirichlet distribution induces variability in mixture proportions while keeping outputs within the simplex, making it a convenient prior for models that combine multiple component distributions.

7.1 Categorical and multinomial distributions

The categorical and multinomial distributions describe outcomes given a fixed probability vector \(p\). The Dirichlet distribution supplies a prior over such probability vectors, enabling Bayesian modeling where uncertainty about \(p\) is explicitly represented.

7.2 Beta distribution and generalizations

As noted for \(K=2\), the Dirichlet distribution reduces to the Beta distribution. More broadly, the Dirichlet family can be viewed as the natural multi-category analogue of Beta priors for binary probability parameters.

7.3 Logistic-normal distribution comparisons

The logistic-normal distribution uses a Gaussian model in log-ratio space and then maps back to the simplex via a softmax transformation. Compared with Dirichlet priors, logistic-normal models can capture more flexible covariance patterns and different tail behaviors, but they typically lack the exact conjugacy enjoyed by the Dirichlet with multinomial likelihoods.

7.4 Dirichlet-multinomial and Polya’s urn

The Dirichlet-multinomial distribution arises by integrating a multinomial likelihood over a Dirichlet prior. The Polya’s urn interpretation provides an intuitive process: repeated sampling tends to reinforce previously drawn categories, reflecting the “pseudo-count” effect embedded in the Dirichlet parameters.

8 Practical Considerations

8.1 Choosing concentration parameters (edge cases)

Selecting \(\alpha_i\) requires balancing prior strength and desired sparsity. Edge cases occur when some \(\alpha_i\) are very small (favoring near-zero components) or when a large disparity among \(\alpha_i\) is used (strongly biasing certain categories). Care is needed because these choices can dramatically affect posterior uncertainty even with modest data.

8.2 Behavior near the simplex boundaries

For \(\alpha_i<1\), the density increases near \(x_i=0\), which can make samples frequently sparse. For \(\alpha_i>1\), the density tends to be flatter near the boundaries and more mass concentrates in the interior. This boundary behavior matters for applications where zeros are plausible or where probability mass should remain spread out.

8.3 Interpreting concentration vs sparsity

Concentration \(\alpha_0\) controls the overall “confidence” in the mean vector. In parallel, whether components are likely to be small is influenced by whether individual \(\alpha_i\) are below or above 1. Thus, two Dirichlet priors can share the same mean but differ substantially in typical sparsity patterns and uncertainty.

8.4 Common pitfalls in implementation and inference

Common issues include:

  • Confusing parameterization conventions for Gamma distributions used in sampling.
  • Using inappropriate optimization settings for \(\boldsymbol{\alpha}\) estimation, leading to poor convergence.
  • Misinterpreting very small \(\alpha_i\) values as “hard zeros,” since they produce strong probability near zero but do not force components to be exactly zero.
  • Numerical instability when computing log densities or normalizers for extreme parameter values.