1. Overview
This section generalizes the univariate normal model to the multivariate case.
We analyze the Bayesian inference for the mean vector θ of a multivariate normal distribution, assuming that the variance–covariance matrix Σ is known.
The results parallel those of the univariate model but introduce important matrix generalizations (e.g., covariance matrices, precision matrices), which are fundamental for Bayesian linear models and multivariate analysis.
2. Model Setup
Let the observed vector $\mathbf{y}$ have $d$ components.
The sampling model is:
$\mathbf{y} \mid \boldsymbol{θ}, \Sigma \sim N_d(\boldsymbol{θ}, \Sigma)$
where:
- $\boldsymbol{θ}$ is a $d \times 1$ mean vector,
- $\Sigma$ is a $d \times d$ covariance matrix (symmetric and positive definite).
For a single observation:
$p(\mathbf{y} \mid \boldsymbol{θ}, \Sigma) \propto |\Sigma|^{-1/2} \exp\!\left[-\frac{1}{2}(\mathbf{y} – \boldsymbol{θ})^T \Sigma^{-1} (\mathbf{y} – \boldsymbol{θ})\right]$
For $n$ independent and identically distributed (i.i.d.) observations $\mathbf{y}_1, \ldots, \mathbf{y}_n$:
$p(\mathbf{y}_1, \ldots, \mathbf{y}_n \mid \boldsymbol{θ}, \Sigma) \propto |\Sigma|^{-n/2} \exp\!\left[-\frac{1}{2}\sum_{i=1}^{n}(\mathbf{y}_i – \boldsymbol{θ})^T \Sigma^{-1} (\mathbf{y}_i – \boldsymbol{θ})\right]$ (3.11)
3. Conjugate Prior for the Mean (Known Σ)
Because the log-likelihood is a quadratic form in $\boldsymbol{θ}$, the conjugate prior for $\boldsymbol{θ}$ is multivariate normal:
$\boldsymbol{θ} \sim N_d(\boldsymbol{θ}_0, \Lambda_0)$
where:
- $\boldsymbol{θ}_0$ = prior mean vector,
- $\Lambda_0$ = prior covariance matrix.
4. Posterior Distribution of θ
The posterior density combines the likelihood (3.11) and the prior:
$p(\boldsymbol{θ} \mid \mathbf{y}, \Sigma) \propto \exp\!\left[-\frac{1}{2}(\boldsymbol{θ} – \boldsymbol{θ}_0)^T \Lambda_0^{-1}(\boldsymbol{θ} – \boldsymbol{θ}_0) -\frac{1}{2}\sum_{i=1}^{n}(\mathbf{y}_i – \boldsymbol{θ})^T \Sigma^{-1} (\mathbf{y}_i – \boldsymbol{θ})\right]$
Completing the square yields another multivariate normal posterior:
$\boldsymbol{θ} \mid \mathbf{y}, \Sigma \sim N_d(\boldsymbol{θ}_n, \Lambda_n)$
with parameters:
$\begin{aligned} \Lambda_n^{-1} &= \Lambda_0^{-1} + n\Sigma^{-1}, \\[4pt] \boldsymbol{θ}_n &= \Lambda_n \left(\Lambda_0^{-1}\boldsymbol{θ}_0 + n\Sigma^{-1}\bar{\mathbf{y}}\right), \end{aligned}$
where $\bar{\mathbf{y}} = \frac{1}{n}\sum_{i=1}^n \mathbf{y}_i$ is the sample mean vector.
5. Interpretation
- The posterior mean $\boldsymbol{θ}_n$ is a weighted average of the prior mean $\boldsymbol{θ}_0$ and the sample mean $\bar{\mathbf{y}}$,
with weights determined by their precision matrices:- Prior precision: $\Lambda_0^{-1}$
- Data precision: $n\Sigma^{-1}$
$\boldsymbol{θ}_n = (\Lambda_0^{-1} + n\Sigma^{-1})^{-1} (\Lambda_0^{-1}\boldsymbol{θ}_0 + n\Sigma^{-1}\bar{\mathbf{y}})$
- The posterior covariance $\Lambda_n$ reflects updated uncertainty:
- $\Lambda_n = (\Lambda_0^{-1} + n\Sigma^{-1})^{-1}$
- The posterior precision adds the prior and data precisions:
- $\Lambda_n^{-1} = \Lambda_0^{-1} + n\Sigma^{-1}$
This mirrors the univariate normal case but with matrix generalization.
6. Marginal and Conditional Posteriors for Subvectors
If $\boldsymbol{θ} = (\boldsymbol{θ}^{(1)}, \boldsymbol{θ}^{(2)})$, then:
- Marginal posterior for $\boldsymbol{θ}^{(1)}$ is multivariate normal:
- $\boldsymbol{θ}^{(1)} \mid \mathbf{y}, \Sigma \sim N(\boldsymbol{θ}_n^{(1)}, \Lambda_n^{(11)})$
- where $\Lambda_n^{(11)}$ is the corresponding submatrix of $\Lambda_n$.
- Conditional posterior for $\boldsymbol{θ}^{(1)} \mid \boldsymbol{θ}^{(2)}, \mathbf{y}$ is also normal:
- $\boldsymbol{θ}^{(1)} \mid \boldsymbol{θ}^{(2)}, \mathbf{y}, \Sigma \sim N\!\left(\boldsymbol{θ}_n^{(1)} + \beta_{1|2}(\boldsymbol{θ}^{(2)} – \boldsymbol{θ}_n^{(2)}), \Lambda_{1|2}\right)$
- where:
- $\begin{aligned} \beta_{1|2} &= \Lambda_n^{(12)}(\Lambda_n^{(22)})^{-1}, \\[4pt] \Lambda_{1|2} &= \Lambda_n^{(11)} – \Lambda_n^{(12)}(\Lambda_n^{(22)})^{-1}\Lambda_n^{(21)}. \end{aligned}$
This expresses how one part of $\boldsymbol{θ}$ depends linearly on another—an idea central to Bayesian linear regression.
7. Posterior Predictive Distribution
For a new observation $\tilde{\mathbf{y}} \sim N_d(\boldsymbol{θ}, \Sigma)$:
$p(\tilde{\mathbf{y}} \mid \mathbf{y}) = \int p(\tilde{\mathbf{y}} \mid \boldsymbol{θ}, \Sigma)\, p(\boldsymbol{θ} \mid \mathbf{y}, \Sigma)\, d\boldsymbol{θ}$
Since both are normal, the predictive distribution is also multivariate normal:
$\tilde{\mathbf{y}} \mid \mathbf{y} \sim N_d(\boldsymbol{θ}_n, \Sigma + \Lambda_n)$
with:
$E[\tilde{\mathbf{y}} \mid \mathbf{y}] = \boldsymbol{θ}_n, \quad \text{Var}[\tilde{\mathbf{y}} \mid \mathbf{y}] = \Sigma + \Lambda_n$
Interpretation:
- $\Sigma$: inherent data variability,
- $\Lambda_n$: posterior uncertainty about the mean $\boldsymbol{θ}$.
8. Noninformative Prior
A flat (noninformative) prior for $\boldsymbol{θ}$ is:
$p(\boldsymbol{θ}) \propto 1$
This corresponds to $|\Lambda_0^{-1}| \to 0$ (infinite prior variance).
The resulting posterior is proper and proportional to the likelihood.
Hence:
$\boldsymbol{θ} \mid \mathbf{y}, \Sigma \sim N_d(\bar{\mathbf{y}}, \Sigma / n)$
This matches the classical sampling distribution of the sample mean under known variance.
9. Summary Table
| Quantity | Formula | Description |
|---|---|---|
| Prior | $\boldsymbol{θ} \sim N_d(\boldsymbol{θ}_0, \Lambda_0)$ | Conjugate prior |
| Posterior | $\boldsymbol{θ} \mid \mathbf{y} \sim N_d(\boldsymbol{θ}_n, \Lambda_n)$ | Updated mean and covariance |
| Posterior mean | $\boldsymbol{θ}_n = (\Lambda_0^{-1} + n\Sigma^{-1})^{-1} (\Lambda_0^{-1}\boldsymbol{θ}_0 + n\Sigma^{-1}\bar{\mathbf{y}})$ | Weighted average |
| Posterior covariance | $\Lambda_n = (\Lambda_0^{-1} + n\Sigma^{-1})^{-1}$ | Reduced uncertainty |
| Predictive mean | $E[\tilde{\mathbf{y}} \mid \mathbf{y}] = \boldsymbol{θ}_n$ | Expected new data |
| Predictive variance | $\Sigma + \Lambda_n$ | Includes both model and parameter uncertainty |
| Noninformative posterior | $N_d(\bar{\mathbf{y}}, \Sigma/n)$ | Flat prior case |
10. Intuitive Summary
- The multivariate normal model generalizes the univariate case by replacing scalar variance with covariance matrices.
- The posterior remains multivariate normal, with parameters updated via matrix-weighted averages of prior and data.
- Each component of $\boldsymbol{θ}$ can be analyzed conditionally or marginally, maintaining normality.
- The posterior predictive distribution accounts for both data variability (Σ) and parameter uncertainty (Λₙ).
- When using a noninformative prior, the posterior for $\boldsymbol{θ}$ equals the sampling distribution of the sample mean.
In short:
The multivariate normal model with known variance provides a mathematically elegant, fully closed-form Bayesian framework. It extends the scalar normal model to multiple dimensions while preserving conjugacy, interpretability, and the ability to derive predictive distributions analytically.