1 Multinomial distribution and likelihood basics

1.1 Definition of the multinomial model

The multinomial model describes the probabilities of obtaining a vector of counts across \(K\) mutually exclusive categories after a fixed number of trials \(n\). If the category probabilities are \(\mathbf{p}=(p_1,\dots,p_K)\) with \(\sum_{i=1}^K p_i=1\), then the random count vector \(\mathbf{X}=(X_1,\dots,X_K)\) satisfies \(X_i\ge 0\) and \(\sum_{i=1}^K X_i=n\). The multinomial distribution assigns probability to each feasible count vector, and this probability becomes the multinomial likelihood when the observed counts are treated as data.

1.2 From probability mass function to likelihood function

For a given observed count vector \(\mathbf{x}=(x_1,\dots,x_K)\) with \(\sum x_i=n\), the multinomial probability mass function (pmf) becomes a likelihood function in the unknown parameters \(\mathbf{p}\). Thus, the multinomial likelihood has the same functional form as the pmf, but interpreted as a function of the parameters rather than the random variable. This shift in viewpoint is central to inference: the data are fixed and the parameter values determine how plausible the data are.

1.3 Relation to the binomial distribution

The binomial distribution can be seen as a two-category special case of the multinomial distribution. When \(K=2\), the count vector can be represented by a single count \(X_1\), with \(X_2=n-X_1\), and the multinomial reduces to the binomial pmf. Consequently, multinomial likelihood generalizes binomial likelihood by allowing more than two outcomes per trial and capturing the joint pattern of counts across categories.

1.4 Notation and assumptions (fixed trials, categorical outcomes)

A standard setup assumes:

  • A fixed number of trials \(n\).
  • Categories that are mutually exclusive and collectively exhaustive.
  • Each trial uses the same category probability vector \(\mathbf{p}\) (no time-varying probabilities unless explicitly modeled through extensions).
  • The observed data are the resulting counts, not the ordering of individual outcomes.

These assumptions determine the structure of the likelihood and the constraints on \(\mathbf{p}\).

2 Likelihood expression and derivation

2.1 Multinomial coefficient and its role

For observed counts \(\mathbf{x}\), the multinomial likelihood includes the multinomial coefficient \[ \frac{n!}{x_1!\cdots x_K!}. \] This factor counts the number of distinct sequences of category labels that produce the same aggregated counts. In likelihood-based inference, the coefficient is constant with respect to \(\mathbf{p}\), so it typically does not affect parameter optimization, though it matters for full probability calculations and for model comparison criteria that use the complete likelihood.

2.2 General likelihood for observed count vectors

The multinomial likelihood for parameters \(\mathbf{p}\) given counts \(\mathbf{x}\) is \[ L(\mathbf{p}\mid \mathbf{x}) \;=\; \frac{n!}{x_1!\cdots x_K!}\prod_{i=1}^K p_i^{x_i}, \] with the constraint \(p_i\ge 0\) and \(\sum_{i=1}^K p_i=1\). The product form reflects independence at the level of each trial given \(\mathbf{p}\), while the coefficient captures combinatorial grouping from sequences to counts.

2.3 Special cases and constraints on parameters

When some categories are impossible under the parameterization (e.g., \(p_i=0\)), the likelihood behavior depends on the corresponding observed count. If \(x_i>0\) while \(p_i=0\), the likelihood becomes zero. If \(x_i=0\), the factor \(p_i^{x_i}\) contributes as \(1\) in the limit, so zero probabilities are compatible with zero observed counts.

Constraints are also important: because probabilities must sum to one, \(\mathbf{p}\) lives on a simplex. In optimization and Bayesian updates, this constraint often motivates reparameterizations or constrained solvers.

2.4 Handling zero counts and numerical stability

Likelihood evaluations are sensitive to underflow because products of many probabilities can become extremely small. A common remedy is to use the log-likelihood, \[ \ell(\mathbf{p}) = \log L(\mathbf{p}\mid \mathbf{x}) = \text{const} + \sum_{i=1}^K x_i \log p_i, \] where “const” denotes terms not depending on \(\mathbf{p}\). For categories with \(x_i=0\), the contribution \(x_i\log p_i\) is treated as \(0\) even if \(\log p_i\) is undefined; computational implementations handle this with conditional logic (e.g., adding terms only when \(x_i>0\)). When \(p_i\) is tiny but nonzero, log computations improve stability.

3 Maximum likelihood estimation (MLE)

3.1 MLE for category probabilities

The maximum likelihood estimator for the category probabilities is \[ \hat{p}_i = \frac{x_i}{n},\quad i=1,\dots,K, \] provided all categories are allowed under the parameter space (i.e., no additional restrictions beyond nonnegativity and summing to one). This result follows from maximizing \(\sum_i x_i\log p_i\) under the simplex constraint.

3.2 Interpretation as normalized empirical frequencies

The MLE has a direct intuitive interpretation: it sets each probability equal to the observed relative frequency. Because the multinomial model is exchangeable across trials given \(\mathbf{p}\), the likelihood is maximized when the model’s probabilities match the empirical composition of the sample.

3.3 Likelihood surface and identifiability

For strictly positive counts, the log-likelihood is strictly concave over the interior of the simplex, producing a single maximizer. If some counts are zero, the likelihood surface becomes flatter along directions that keep those probabilities at or near zero; the data still identify the probabilities for observed categories, but boundaries can introduce non-uniqueness in limiting solutions when additional constraints are absent.

Identifiability is usually straightforward because the multinomial likelihood depends on \(\mathbf{p}\) only through the individual components \(p_i\). Complications arise in extended models (e.g., mixtures or regressions) where different parameter settings can yield similar induced probabilities.

3.4 Comparing nested models via likelihood ratios

Likelihood ratio tests can compare nested hypotheses about the parameterization (for instance, constraints that force subsets of probabilities to be equal, or models that reduce the degrees of freedom). The test statistic typically uses the ratio of maximized likelihoods under the null and alternative models. For multinomial data, the log-likelihood form above makes it practical to compute such ratios, though asymptotic assumptions must be checked in finite samples.

4 Bayesian treatment

4.1 Dirichlet prior and conjugacy

A Dirichlet prior is conjugate to the multinomial likelihood. With a prior \[ \mathbf{p} \sim \text{Dirichlet}(\boldsymbol{\alpha}),\quad \alpha_i>0, \] the posterior distribution remains Dirichlet after observing counts \(\mathbf{x}\). Conjugacy yields closed-form expressions for posterior quantities and makes Bayesian computation efficient.

4.2 Posterior distribution for category probabilities

Given counts \(\mathbf{x}\), the posterior is \[ \mathbf{p}\mid \mathbf{x} \sim \text{Dirichlet}(\boldsymbol{\alpha}+\mathbf{x}), \] meaning the posterior concentration parameters are updated by adding observed counts componentwise. This update blends prior beliefs (via \(\boldsymbol{\alpha}\)) with data evidence (via \(\mathbf{x}\)), with the relative impact determined by the magnitude of the prior concentration.

4.3 Posterior predictive distribution

For predicting the next categorical outcome, the posterior predictive probability of category \(i\) is \[ \Pr(\text{next is } i\mid \mathbf{x}) = \frac{\alpha_i + x_i}{\sum_{j=1}^K (\alpha_j + x_j)}. \] More generally, for predicting a future count vector from another multinomial experiment with the same category structure, the posterior predictive distribution becomes a Dirichlet-multinomial form, capturing uncertainty about \(\mathbf{p}\).

4.4 MAP estimation under Dirichlet hyperparameters

The maximum a posteriori (MAP) estimate under a Dirichlet prior has a simple form when \(\alpha_i>1\) for all \(i\) (so the mode lies in the interior): \[ \hat{p}_i^{\text{MAP}} = \frac{\alpha_i + x_i - 1}{\sum_{j=1}^K (\alpha_j + x_j - 1)}. \] When some \(\alpha_i \le 1\), the posterior mode may occur on the boundary, reflecting the prior’s tendency to favor sparse probability vectors. In such cases, the MAP may differ qualitatively from the posterior mean.

5 Computational considerations

5.1 Working with log-likelihoods

Most numerical workflows compute the log-likelihood rather than the raw likelihood to avoid underflow. For optimization, gradients are also derived from the log-likelihood since additive constants from the coefficient term drop out when differentiating.

5.2 Normalization and constant terms

Because the multinomial coefficient does not depend on \(\mathbf{p}\), it can be omitted in gradient-based optimization. However, it must be included when computing absolute likelihood values for model comparison. When using log-likelihoods, it is common to compute the coefficient term using log-factorials (e.g., via stable special functions or precomputed tables) to handle large \(n\).

5.3 Gradient and optimization for constrained parameters

For basic multinomial MLE, the closed-form solution eliminates the need for numerical optimization. For variants with constraints or reparameterizations—such as models enforcing equality among some probabilities, or frameworks where \(\mathbf{p}\) is parameterized through logits—numerical optimization is typical. Gradients must respect the simplex constraint. Approaches include:

  • Reparameterizing with softmax transforms of unconstrained variables.
  • Using constrained optimizers that enforce \(p_i\ge 0\) and \(\sum p_i=1\).
  • Employing projected methods where iterates are mapped back to the simplex.

5.4 Implementation tips and common pitfalls

Common issues include:

  • Treating \(0\log 0\) incorrectly when \(x_i=0\). In theory this term contributes \(0\), but implementations must avoid producing NaNs.
  • Allowing probabilities to become exactly zero in gradient methods, which can stall optimization when using log terms.
  • Confusing the multinomial likelihood with the multinomial coefficient; only the \(\prod p_i^{x_i}\) part depends on \(\mathbf{p}\).
  • Forgetting the normalization constraint when using custom parameter updates.

Robust code typically uses log-space computations, checks constraints, and includes explicit handling for zero counts.

6 Model evaluation and diagnostics

6.1 Goodness-of-fit concepts

Goodness-of-fit checks aim to assess whether the predicted category probabilities are compatible with observed counts. In multinomial contexts, deviations appear in the distribution of counts across categories rather than in any single ordered sequence. Evaluations often rely on likelihood-based measures, discrepancy statistics, or predictive performance.

6.2 Residuals and discrepancy measures for multinomial data

Residual ideas differ from continuous regression settings because observations are counts summing to a fixed total. Useful diagnostics include comparing observed and expected counts \(x_i\) and \(n p_i\), and computing standardized or deviance-like measures. For example, a chi-square-like statistic can be built from \((x_i - n p_i)^2/(n p_i)\) when expected counts are sufficiently large, while deviance or likelihood-based discrepancies can be preferred in smaller samples.

6.3 Predictive checks and cross-validation

Posterior predictive checks evaluate whether simulated counts from the fitted model resemble the observed counts. One common practice is to generate replicated datasets using the fitted parameters (or posterior draws) and compare summary statistics, such as the largest category count, entropy, or log-likelihood. Cross-validation can also be used by fitting the model on training folds and scoring held-out counts via the predictive likelihood.

6.4 Calibration of predicted category probabilities

Calibration assesses whether predicted probabilities match empirical frequencies. In multinomial settings, calibration can be checked by comparing predicted and realized proportions across categories or by assessing how often high-probability categories occur. With Bayesian models, calibration can also be evaluated using posterior predictive distributions: if predictive intervals for category probabilities or counts consistently miss the observed outcomes, the model’s probabilistic assumptions may be misaligned.

7.1 Multinomial with varying totals (generalized counts perspective)

The standard multinomial assumes a fixed total \(n\). When totals vary by observation (for instance, across different surveys or documents), one can treat each observation as its own multinomial experiment with its own \(n\), then combine likelihoods across observations. This yields a generalized likelihood for multiple count vectors, each governed by the same or related probability parameters, depending on the modeling goal.

7.2 Multinomial logistic regression (linking probabilities to covariates)

Multinomial logistic regression generalizes the multinomial distribution by modeling how probabilities depend on covariates. The model expresses \(p_i\) through a softmax over linear predictors, so category probabilities vary with explanatory variables. The multinomial likelihood becomes the training objective, and estimation proceeds via maximizing total log-likelihood over all observations.

7.3 Hierarchical multinomial models

Hierarchical formulations introduce random effects or group-level parameters, allowing category probabilities to vary across groups while sharing statistical strength. For example, one may model group-specific probability vectors as drawn from a higher-level Dirichlet distribution. Such structures address overdispersion relative to the basic multinomial and improve robustness when some groups have limited sample sizes.

7.4 Connection to categorical distributions and mixture components

A multinomial model can be viewed as the distribution of counts generated by repeated draws from a categorical distribution. In mixture settings, each observation’s category probabilities may itself be a mixture over latent components, leading to mixture-of-multinomials or related latent-variable models. In these cases, the simple closed-form likelihood is replaced by marginalization over latent assignments, and inference typically relies on methods such as expectation-maximization or Markov chain Monte Carlo.