1 Categorical Distribution Basics
1.1 Definition and interpretation
A categorical distribution is a probability distribution defined on a finite set of mutually exclusive outcomes. One outcome (one category) is selected per draw. If the set of categories is indexed by \(1,2,\dots,K\), the distribution specifies the probability \(p_k\) assigned to each category \(k\). The key assumption is exclusivity: exactly one category occurs for each trial.
In practice, the categorical distribution appears whenever a system must choose among a finite set of labels, such as classifying an observation into one of several classes or assigning a component label to a data point.
1.2 Support (outcomes) and probability vector
The support is the collection of categories \(\{1,2,\dots,K\}\) (or an equivalent labeling of categories). The distribution can be represented by a probability vector \[ \mathbf{p}=(p_1,\dots,p_K), \] where each component \(p_k\) is the probability of category \(k\).
The defining constraint is that the probabilities are nonnegative and collectively exhaust all possibilities, so the components sum to one.
1.3 Probability mass function (PMF)
The probability mass function gives the probability of each outcome value: \[ \Pr(X=k)=p_k,\quad k\in\{1,\dots,K\}. \] Here \(X\) denotes the categorical random variable. The PMF is discrete by nature because the support is finite.
1.4 Relationship to other distributions
1.4.1 Bernoulli as a special case (two categories)
When there are exactly two categories, a categorical distribution reduces to a Bernoulli distribution. For example, with categories \(\{0,1\}\), the categorical parameters are \((1-p,\,p)\), and the random variable can be interpreted as “success” with probability \(p\).
More generally, categorical distributions can be viewed as a direct multi-category extension of Bernoulli trials.
2 Parameterization and Properties
2.1 Parameter constraints (nonnegativity and sum-to-one)
The parameters must satisfy:
- \(p_k \ge 0\) for each category \(k\),
- \(\sum_{k=1}^K p_k = 1\).
These constraints ensure that the PMF is valid. In estimation and learning, this leads to parameterizations that automatically enforce the constraints, such as using a softmax mapping from unconstrained real numbers to nonnegative normalized probabilities.
2.2 Common notations (p, π, θ)
Different fields and software packages use different symbols:
- \(p\) or \(\mathbf{p}\) for the probability vector,
- \(\boldsymbol{\pi}\) in mixture-model or Markov contexts (often interpreted as “mixing proportions” or “state probabilities”),
- \(\boldsymbol{\theta}\) when emphasizing model parameters linked to a categorical distribution.
Although notation changes, the underlying object is the same: a set of category probabilities summing to one.
2.3 Expectation of categorical variables
The expectation depends on how categories are encoded. If categories correspond directly to numeric values \(v_k\), then \[ \mathbb{E}[v(X)] = \sum_{k=1}^K v_k\, p_k. \] If instead one uses one-hot encoding (indicator variables), expectations align with the probabilities: the expected value of the indicator for category \(k\) equals \(p_k\).
2.4 Variance and second moments
As with expectation, variance depends on the encoding. For a numeric-valued mapping \(v_k\), \[ \mathrm{Var}(v(X))=\sum_{k=1}^K v_k^2 p_k - \left(\sum_{k=1}^K v_k p_k\right)^2. \] For one-hot indicators \(I_k=\mathbf{1}\{X=k\}\), the second moments satisfy \(I_k^2=I_k\), giving \[ \mathbb{E}[I_k]=p_k,\quad \mathrm{Var}(I_k)=p_k(1-p_k), \] and for \(k\neq \ell\), \(I_k I_\ell=0\), implying \(\mathrm{Cov}(I_k,I_\ell)=-p_k p_\ell\).
2.5 Independence considerations in repeated trials
Repeated draws from the same categorical distribution are independent if each draw is generated without reference to previous outcomes. If \(X_1,\dots,X_n\) are i.i.d. categorical draws with probabilities \(\mathbf{p}\), then for any categories \(k_1,\dots,k_n\), \[ \Pr(X_1=k_1,\dots,X_n=k_n)=\prod_{i=1}^n p_{k_i}. \] If dependence exists (e.g., via a Markov process), the joint behavior changes and the model may no longer be representable as i.i.d. categorical draws.
3 Sampling and Computation
3.1 Sampling from categorical probabilities
Sampling converts the probability vector \(\mathbf{p}\) into a category draw. A standard approach uses cumulative probabilities: form the cumulative sum \[ F(k)=\sum_{i=1}^k p_i, \] then draw a uniform random number \(U\sim\mathrm{Uniform}(0,1)\) and select the smallest \(k\) such that \(U \le F(k)\). This yields exactly \(\Pr(X=k)=p_k\).
3.2 Inverse transform / cumulative method
The cumulative method is an instance of the inverse transform technique. By mapping \(U\) through the inverse CDF of the discrete distribution, one obtains the correct discrete outcome probabilities. Because the distribution is finite, computing the cumulative sums and then searching for the threshold is typically efficient.
When \(K\) is large, alternative sampling strategies (such as alias tables) can reduce the cost per draw, though the categorical definition itself remains unchanged.
3.3 Efficient implementation details
Practical implementations often precompute cumulative probabilities for repeated sampling with the same \(\mathbf{p}\). Common strategies include:
- Storing the cumulative array \(F(1),\dots,F(K)\) and using binary search to locate the matching category.
- Handling category indexing consistently with the target language’s array conventions.
- Using vectorized operations when sampling many draws at once.
In learning pipelines, categorical sampling can also be used for stochastic decisions, such as generating from a learned predictive distribution.
3.4 Numerical stability (normalization and underflow)
In floating-point computations, the probabilities may not sum to exactly one due to rounding, or they may be extremely small. Stability measures include:
- Renormalizing: if \(s=\sum_k p_k\), replace \(p_k\leftarrow p_k/s\) when \(s\) is close to one but not exact.
- Guarding against underflow: when probabilities are derived from log-probabilities, exponentiating may underflow; using a “log-sum-exp” normalization step helps.
- Ensuring the cumulative sum reaches one within tolerance, for example by forcing the final cumulative entry to be exactly one after normalization.
These steps prevent sampling logic from failing due to small numerical discrepancies.
4 Likelihood, Estimation, and Learning
4.1 Maximum likelihood estimation (MLE)
Given observed categorical labels \(x_1,\dots,x_n\) from categories \(\{1,\dots,K\}\), the log-likelihood of parameters \(\mathbf{p}\) is \[ \log L(\mathbf{p})=\sum_{i=1}^n \log p_{x_i}. \] Let \(n_k\) be the count of observations in category \(k\). Maximizing the likelihood under the constraints \(\sum_k p_k=1\) and \(p_k\ge 0\) yields \[ \hat{p}_k=\frac{n_k}{n}, \] the empirical relative frequencies.
4.2 Relation to counts and the multinomial distribution
If one draws \(n\) i.i.d. samples from a categorical distribution with probabilities \(\mathbf{p}\), then the vector of counts \((n_1,\dots,n_K)\) follows a multinomial distribution. The categorical MLE corresponds to normalizing those counts: \(\hat{p}_k=n_k/n\).
Thus, categorical models at the “single draw” level connect naturally to multinomial likelihoods at the “batch counts” level.
4.3 Bayesian estimation with Dirichlet priors
A common Bayesian treatment places a Dirichlet prior on \(\mathbf{p}\): \[ \mathbf{p}\sim \mathrm{Dir}(\boldsymbol{\alpha}), \] with concentration parameters \(\alpha_k>0\). After observing counts \(n_k\), the posterior is also Dirichlet: \[ \mathbf{p}\mid \text{data}\sim \mathrm{Dir}(\alpha_1+n_1,\dots,\alpha_K+n_K). \] The posterior mean becomes \[ \mathbb{E}[p_k\mid \text{data}] = \frac{\alpha_k+n_k}{\sum_{j=1}^K(\alpha_j+n_j)}. \] This form provides smoothing and avoids zero-probability estimates when some categories are not observed.
4.4 Posterior predictive distribution
The posterior predictive probability for the next draw is obtained by integrating over \(\mathbf{p}\). Under the Dirichlet model, \[ \Pr(X_{\text{new}}=k\mid \text{data})=\frac{\alpha_k+n_k}{\sum_{j=1}^K(\alpha_j+n_j)}. \] This is the Bayesian counterpart to the empirical frequency estimate, with prior mass influencing predictions, especially for small datasets.
5 Cross-Entropy and Log-Likelihood
5.1 Negative log-likelihood for categorical outcomes
For a single observed category \(x\), the negative log-likelihood is \[ -\log p_x. \] For a dataset, summing over observations gives an objective that penalizes assigning low probability to the realized category. This quantity is widely used as a training loss in probabilistic classification.
5.2 Cross-entropy with one-hot targets
If a model predicts a categorical distribution \(\mathbf{q}=(q_1,\dots,q_K)\) and the true label is encoded as a one-hot vector \(\mathbf{y}\) (where \(y_k=1\) for the correct category and \(0\) otherwise), the cross-entropy is \[ H(\mathbf{y},\mathbf{q})=-\sum_{k=1}^K y_k\log q_k. \] Because exactly one \(y_k\) equals one, this reduces to \(-\log q_{x}\), matching the negative log-likelihood for that labeled instance.
5.3 Perplexity-style interpretations (qualitative)
In some applications, particularly language modeling, one exponentiates an averaged cross-entropy to obtain a “perplexity” measure. Qualitatively, higher perplexity corresponds to more uncertainty (the model’s predicted distribution is more spread out), while lower perplexity indicates sharper, better-calibrated predictions.
Even outside language modeling, the averaged cross-entropy can be interpreted as an uncertainty-adjusted scoring rule for categorical predictions.
6 Entropy and Information Measures
6.1 Shannon entropy of a categorical distribution
The Shannon entropy quantifies uncertainty for the categorical distribution \(\mathbf{p}\): \[ H(\mathbf{p})=-\sum_{k=1}^K p_k\log p_k. \] Entropy is maximized when probabilities are uniform and minimized when one category has probability one. The base of the logarithm determines the unit (e.g., nats for natural log, bits for log base 2).
6.2 KL divergence between categoricals
The Kullback–Leibler (KL) divergence measures how one distribution diverges from another. For two categoricals \(\mathbf{p}\) and \(\mathbf{q}\), \[
| D_{\mathrm{KL}}(\mathbf{p}\,\|\,\mathbf{q})=\sum_{k=1}^K p_k\log\frac{p_k}{q_k}. |
|---|
\] It is nonnegative and equals zero only when the distributions match on the support where \(p_k>0\). If \(q_k=0\) while \(p_k>0\), the divergence is infinite.
6.3 Practical implications for model comparison
In many learning workflows, KL divergence appears implicitly through differences in log-likelihood or cross-entropy. When comparing two candidate predictive distributions for the same data-generating process, the distribution that assigns higher probability mass to the true outcomes will typically yield lower expected log-loss, which is closely related to smaller divergence measures.
KL divergence is also useful for analyzing how much information is lost when approximating a target distribution with a simpler one.
7 Modeling Uses and Extensions
7.1 Classification outputs and softmax connection
A common pipeline in multi-class classification models outputs unnormalized scores for each class (often called logits). Applying the softmax function converts these scores into a categorical probability vector: \[ q_k=\frac{e^{z_k}}{\sum_{j=1}^K e^{z_j}}. \] The resulting \(\mathbf{q}\) defines a categorical distribution over classes, and the training loss commonly corresponds to cross-entropy with one-hot targets.
This provides a principled probabilistic interpretation of model outputs: predicted probabilities directly reflect the categorical distribution.
7.2 Mixture models and latent category assignments
In mixture models, each data point is associated with a latent component index that can be modeled as a categorical variable. The mixing weights define the probabilities of selecting each component, while component-specific parameters govern the distribution of observed data conditioned on the latent assignment.
Learning often alternates between estimating component responsibilities (posterior categorical probabilities over latent labels) and updating component parameters.
7.3 Markov transitions as categorical draws
Markov models use categorical distributions to represent transitions between states. Given the current state, the next state is drawn from a categorical distribution whose probabilities are given by the corresponding row of the transition matrix.
This framework makes the categorical distribution a building block for discrete-time stochastic processes on a finite state space.
7.4 Higher-level generalizations (multinomial, mixture of categoricals)
The categorical distribution generalizes to broader discrete multivariate settings. The multinomial distribution describes counts from multiple i.i.d. categorical draws, while mixtures of categoricals introduce additional latent structure that mixes several categorical distributions.
These extensions preserve the core idea: discrete outcomes governed by probability vectors and weighted selection mechanisms.
8 Worked Examples
8.1 Simple two-category example
Consider a categorical distribution with two categories \(\{0,1\}\) and probabilities \((0.25, 0.75)\). If \(X\) denotes the outcome, then \[ \Pr(X=0)=0.25,\quad \Pr(X=1)=0.75. \] The entropy is \[ H=-0.25\log(0.25)-0.75\log(0.75), \] and the negative log-likelihood for observing \(X=1\) equals \(-\log(0.75)\).
8.2 Multi-category probability calculation
Let \(K=4\) and probabilities be \(\mathbf{p}=(0.1,0.2,0.3,0.4)\). If a category \(3\) (indexing from 1) corresponds to the realized outcome, the probability of that outcome is simply \(p_3=0.3\).
For an event that includes multiple categories, such as \(X\in\{2,4\}\), the probability is the sum: \[ \Pr(X\in\{2,4\})=p_2+p_4=0.2+0.4=0.6. \]
8.3 Estimating probabilities from observed labels
Suppose observations over \(K=3\) categories yield counts \((n_1,n_2,n_3)=(2,5,3)\) for a total \(n=10\). The MLE estimate is \[ \hat{p}_1=\frac{2}{10}=0.2,\quad \hat{p}_2=\frac{5}{10}=0.5,\quad \hat{p}_3=\frac{3}{10}=0.3. \] If a Dirichlet prior with parameters \(\boldsymbol{\alpha}=(1,1,1)\) is used (uniform prior), the posterior predictive for the next draw becomes \[ \Pr(X_{\text{new}}=k\mid \text{data})=\frac{1+n_k}{3+10}=\frac{1+n_k}{13}, \] yielding probabilities \(\frac{3}{13},\frac{6}{13},\frac{4}{13}\).
8.4 Comparing two distributions with KL divergence
Let \(\mathbf{p}=(0.7,0.3)\) and \(\mathbf{q}=(0.6,0.4)\) for a two-category case. The KL divergence from \(\mathbf{p}\) to \(\mathbf{q}\) is \[
| D_{\mathrm{KL}}(\mathbf{p}\,\|\,\mathbf{q})=0.7\log\frac{0.7}{0.6}+0.3\log\frac{0.3}{0.4}. |
|---|
\] This value is nonnegative and becomes zero only if \(p_k=q_k\) for both categories. Numerically it reflects how much extra log-loss one would incur when using \(\mathbf{q}\) instead of \(\mathbf{p}\) to model outcomes drawn from \(\mathbf{p}\).