1. From Binomial to Multinomial Models
The binomial distribution (two possible outcomes) can be extended to the multinomial distribution, which allows k possible outcomes.
If we observe nnn categorical outcomes classified into kkk categories,
and define the vector of counts:
$\mathbf{y} = (y_1, y_2, \ldots, y_k)$
then the multinomial likelihood is:
$p(\mathbf{y} \mid \boldsymbol{θ}) \propto \prod_{j=1}^{k} θ_j^{y_j}$
where:
$\sum_{j=1}^{k} θ_j = 1, \quad \text{and} \quad \sum_{j=1}^{k} y_j = n$
Here $θ_j$ is the probability of outcome $j$, and $y_j$ is the observed count.
2. The Dirichlet Prior (Conjugate to the Multinomial)
The conjugate prior for $\boldsymbol{θ} = (θ_1, θ_2, \ldots, θ_k)$ is the Dirichlet distribution:
$p(\boldsymbol{θ} \mid \boldsymbol{α}) \propto \prod_{j=1}^{k} θ_j^{α_j – 1}$
where all $θ_j \ge 0$ and $\sum_{j=1}^{k} θ_j = 1$.
Each hyperparameter $α_j > 0$ controls the prior strength for category $j$.
Thus, $\boldsymbol{θ} \sim \text{Dirichlet}(α_1, α_2, \ldots, α_k)$.
3. Posterior Distribution
Given the likelihood and prior, the posterior is also Dirichlet (conjugacy property):
$\boldsymbol{θ} \mid \mathbf{y} \sim \text{Dirichlet}(α_1 + y_1, α_2 + y_2, \ldots, α_k + y_k)$
Each parameter update corresponds to adding the observed counts to the prior pseudocounts.
4. Interpretation of ααα Parameters
- The prior parameter $α_j – 1$ can be interpreted as the number of prior “pseudo-observations” in category $j$.
- The total prior sample size is $α_0 = \sum_{j=1}^{k} α_j$.
For example:
- $α_j = 1$: uniform prior (all categories equally likely).
- $α_j = 0$: improper “flat” prior on the log scale (noninformative in log-probability).
If each $y_j > 0$, the posterior is proper even with $α_j = 0$.
5. Noninformative Dirichlet Priors
There are several ways to represent noninformative priors:
| Prior Choice | Interpretation |
|---|---|
| $α_j = 1$ | Uniform prior over the simplex (equal weight to all $θ$ vectors) |
| $α_j = 0$ | Improper prior, uniform on log($θ_j$) scale |
| $α_j = 1/2$ | Jeffreys’ prior (invariant under reparameterization) |
Each choice affects the posterior only slightly for large nnn.
6. Example — Pre-election Polling
In October 1988, a CBS News survey asked 1447 U.S. adults which presidential candidate they supported.
The responses were:
| Category | Count (yjy_jyj) |
|---|---|
| Bush ($y_1$) | 727 |
| Dukakis ($y_2$) | 583 |
| Other/No opinion ($y_3$) | 137 |
Total: $n = 1447$.
The multinomial model:
$(y_1, y_2, y_3) \sim \text{Multinomial}(n = 1447; θ_1, θ_2, θ_3)$
with
$\sum_{j=1}^{3} θ_j = 1$
7. Prior and Posterior for the Polling Example
With a uniform Dirichlet prior ($α_1 = α_2 = α_3 = 1$):
$p(\boldsymbol{θ}) = \text{Dirichlet}(1, 1, 1)$
The posterior becomes:
$\boldsymbol{θ} \mid \mathbf{y} \sim \text{Dirichlet}(728, 584, 138)$
since
$y_j + α_j = (727 + 1, 583 + 1, 137 + 1)$
8. Posterior Simulation and Estimation
A quantity of interest is the difference in population proportions:
$Δ = θ_1 – θ_2$
representing the difference in support for Bush vs. Dukakis.
Analytically integrating $p(Δ \mid \mathbf{y})$ is difficult, so we use Monte Carlo simulation:
- Draw 1000 samples of $(θ_1, θ_2, θ_3)$ from Dirichlet(728, 584, 138).
- Compute $Δ = θ_1 – θ_2$ for each draw.
In these simulations, all 1000 samples had $θ_1 > θ_2$,
implying:
$P(θ_1 > θ_2 \mid \mathbf{y}) > 0.999$
Hence, the posterior probability that Bush had more support than Dukakis exceeds 99.9%.
9. Notes on Survey Design
The analysis above assumes simple random sampling.
However, the actual CBS survey used stratified sampling, which introduces dependence among observations.
A more accurate analysis (accounting for stratification and weights) is discussed later.
10. High-dimensional Extensions
When there are many categories (large $k$), the multinomial model becomes less informative unless additional structure is introduced:
- Hierarchical priors: model parameters $θ_j$ as drawn from a common hyperprior.
- Loglinear models: impose relationships between categorical variables.
Such structures allow sharing of statistical strength across categories and improve interpretability.
11. Summary Table
| Concept | Formula / Description |
|---|---|
| Likelihood | $p(\mathbf{y} \mid \boldsymbol{θ}) \propto \prod_{j=1}^{k} θ_j^{y_j}$ |
| Prior | $p(\boldsymbol{θ}) = \text{Dirichlet}(α_1, \ldots, α_k)$ |
| Posterior | $p(\boldsymbol{θ} \mid \mathbf{y}) = \text{Dirichlet}(α_1 + y_1, \ldots, α_k + y_k)$ |
| Posterior mean | $E(θ_j \mid \mathbf{y}) = \frac{α_j + y_j}{α_0 + n}$, where $α_0 = \sum α_j$ |
| Example posterior | Dirichlet(728, 584, 138) |
| Key estimand | $θ_1 – θ_2$ (difference in support) |
12. Intuitive Summary
- The multinomial distribution generalizes the binomial to more than two outcomes.
- Its conjugate prior is the Dirichlet, which is flexible and easy to update.
- The posterior is again Dirichlet, obtained by simply adding observed counts to prior pseudocounts.
- The parameters $α_j$ represent prior knowledge or pseudodata.
- Bayesian inference for proportions (and their differences) can be done easily through Dirichlet sampling.
- In real surveys, stratification or weighting should be incorporated for accuracy.
In short:
The multinomial–Dirichlet model is a powerful and elegant Bayesian framework for categorical data, where inference about proportions or category differences follows directly from simple count updates and simulation-based summaries.