Multivariate Normal with Unknown Mean and Variance

1. Overview

This model generalizes the univariate normal with unknown mean and variance to the multivariate case.
The unknown parameters are:

• The mean vector $\boldsymbol{θ}$ (dimension $d \times 1$)
• The covariance matrix $\Sigma$ (dimension $d \times d$, symmetric and positive definite)

The conjugate prior family for $(\boldsymbol{θ}, \Sigma)$ is the Normal–Inverse–Wishart (NIW) distribution, which extends the Normal–Inverse–χ² model used in the univariate case.

---

2. Conjugate Prior: The Normal–Inverse–Wishart Family

The NIW prior is parameterized by four hyperparameters:

$(\boldsymbol{θ}_0, \, \Lambda_0 / \kappa_0; \, \nu_0, \, \Lambda_0)$

where:

• $\boldsymbol{θ}_0$​: prior mean vector
• $\kappa_0$​: prior “sample size” for the mean
• $\nu_0$​: prior degrees of freedom for $\Sigma$
• $\Lambda_0$​: prior scale matrix for $\Sigma$

---

3. Prior Structure

The hierarchical form of the prior is:

$\begin{aligned} \Sigma &\sim \text{Inv-Wishart}_{\nu_0}(\Lambda_0^{-1}), \\ \boldsymbol{θ} \mid \Sigma &\sim N_d(\boldsymbol{θ}_0, \Sigma / \kappa_0). \end{aligned}$

This implies the joint prior density:

$p(\boldsymbol{θ}, \Sigma) \propto |\Sigma|^{-(\nu_0 + d)/2 – 1} \exp\!\left[-\frac{1}{2}\operatorname{tr}(\Lambda_0 \Sigma^{-1}) -\frac{\kappa_0}{2}(\boldsymbol{θ} – \boldsymbol{θ}_0)^T \Sigma^{-1}(\boldsymbol{θ} – \boldsymbol{θ}_0)\right]$

Interpretation of hyperparameters:

• $\boldsymbol{θ}_0$​: prior mean belief for $\boldsymbol{θ}$
• $\kappa_0$​: controls how tightly $\boldsymbol{θ}$ is concentrated around $\boldsymbol{θ}_0$
• $\nu_0$​: controls certainty about $\Sigma$
• $\Lambda_0$​: controls scale and orientation of the covariance

---

4. Posterior Distribution

Given data $\{\mathbf{y}_1, \dots, \mathbf{y}_n\}$ with sample mean $\bar{\mathbf{y}}$​ and sample covariance matrix $S = \sum_{i=1}^n (\mathbf{y}_i – \bar{\mathbf{y}})(\mathbf{y}_i – \bar{\mathbf{y}})^T,$

the posterior is also Normal–Inverse–Wishart:

$(\boldsymbol{θ}, \Sigma) \mid \mathbf{y} \sim \text{NIW}(\boldsymbol{θ}_n, \Lambda_n / \kappa_n; \nu_n, \Lambda_n)$

where the updated parameters are:

$\begin{aligned} \kappa_n &= \kappa_0 + n, \\[4pt] \nu_n &= \nu_0 + n, \\[4pt] \boldsymbol{θ}_n &= \frac{\kappa_0 \boldsymbol{θ}_0 + n \bar{\mathbf{y}}}{\kappa_0 + n}, \\[6pt] \Lambda_n &= \Lambda_0 + S + \frac{\kappa_0 n}{\kappa_0 + n}(\bar{\mathbf{y}} – \boldsymbol{θ}_0)(\bar{\mathbf{y}} – \boldsymbol{θ}_0)^T. \end{aligned}$

---

5. Intuitive Meaning of Posterior Updates

• $\boldsymbol{θ}_n$​: weighted average of prior mean and data mean
• $\Lambda_n$​: combines prior scale, sample covariance, and prior–data discrepancy
• $\nu_n$​: increases with nnn, representing growing information about $\Sigma$
• $\kappa_n$​: reflects the total “effective sample size” for $\boldsymbol{θ}$

---

6. Marginal and Predictive Distributions

(a) Marginal posterior of the mean $\boldsymbol{θ}$:

After integrating out $\Sigma$:

$\boldsymbol{θ} \mid \mathbf{y} \sim t_{\nu_n – d + 1}\!\left(\boldsymbol{θ}_n, \frac{\Lambda_n}{\kappa_n (\nu_n – d + 1)}\right)$

This is a multivariate Student-t distribution with:

• mean $\boldsymbol{θ}_n$​,
• scale $\Lambda_n / [\kappa_n (\nu_n – d + 1)]$,
• degrees of freedom $\nu_n – d + 1$.

---

(b) Posterior predictive distribution for new observation $\tilde{\mathbf{y}}$​:

Integrating over both $\boldsymbol{θ}$ and $\Sigma$:

$\tilde{\mathbf{y}} \mid \mathbf{y} \sim t_{\nu_n – d + 1}\!\left( \boldsymbol{θ}_n, \frac{(\kappa_n + 1)\Lambda_n}{\kappa_n (\nu_n – d + 1)} \right)$

This distribution accounts for both uncertainty in the mean and variability in the covariance.

---

7. Posterior Sampling Procedure

To simulate from the posterior $p(\boldsymbol{θ}, \Sigma \mid \mathbf{y})$:

1. Draw $\Sigma \sim \text{Inv-Wishart}_{\nu_n}(\Lambda_n^{-1})$.
2. Given $\Sigma$, draw $\boldsymbol{θ} \sim N_d(\boldsymbol{θ}_n, \Sigma / \kappa_n)$.

To generate a predictive sample $\tilde{\mathbf{y}}$​:

1. Use the sampled $\boldsymbol{θ}, \Sigma$,
2. Draw $\tilde{\mathbf{y}} \sim N_d(\boldsymbol{θ}, \Sigma)$.

---

8. Noninformative Prior Options

1. Inverse-Wishart with $\nu_0 = d + 1$:
   • $\Sigma \sim \text{Inv-Wishart}_{d+1}(I),$ yields approximately uniform priors for correlations.
2. Jeffreys’ prior (improper noninformative prior):
   • $p(\boldsymbol{θ}, \Sigma) \propto |\Sigma|^{-(d + 1)/2}.$
   • Equivalent to the limit as $\kappa_0 \to 0, \, \nu_0 \to -1, \, |\Lambda_0| \to 0$.
   • The posterior then becomes:
     • $\begin{aligned} \Sigma \mid \mathbf{y} &\sim \text{Inv-Wishart}_{n-1}(S^{-1}), \\ \boldsymbol{θ} \mid \Sigma, \mathbf{y} &\sim N_d(\bar{\mathbf{y}}, \Sigma / n). \end{aligned}$.​
   • The marginal posterior for $\boldsymbol{θ}$ is:
     • $\boldsymbol{θ} \mid \mathbf{y} \sim t_{n – d}\!\left(\bar{\mathbf{y}}, \frac{S}{n(n – d)}\right)$.
   • This corresponds to the classical sampling result when both $\boldsymbol{θ}$ and $\Sigma$are unknown.

---

9. Scaled Inverse–Wishart Model

A more flexible alternative introduces separate scaling parameters for each variable:

$Sigma = \operatorname{Diag}(\boldsymbol{ξ}) \, \Sigma_η \, \operatorname{Diag}(\boldsymbol{ξ})$

where:

• $\Sigma_η$ has an inverse-Wishart prior, e.g. $\text{Inv-Wishart}_{d+1}(I)$,
• $\boldsymbol{ξ}$ are scale parameters given weakly informative priors.

This decouples variances from correlations, allowing for weak or uniform priors on correlations while keeping flexible variance priors.
This approach is particularly useful in hierarchical regression models.

---

10. Summary Table

Component	Formula / Distribution	Interpretation
Prior for $\Sigma$	$\text{Inv-Wishart}_{\nu_0}(\Lambda_0^{-1})$	Prior on covariance matrix
Prior for $\boldsymbol{θ} \mid \Sigma$	$N_d(\boldsymbol{θ}_0, \Sigma / \kappa_0)$	Prior mean conditional on variance
Posterior $\Sigma \mid \mathbf{y}$	$\text{Inv-Wishart}_{\nu_n}(\Lambda_n^{-1})$	Updated covariance
Posterior $\boldsymbol{θ} \mid \Sigma, \mathbf{y}$	$N_d(\boldsymbol{θ}_n, \Sigma / \kappa_n)$	Updated mean
Marginal $\boldsymbol{θ} \mid \mathbf{y}$	$t_{\nu_n – d + 1}(\boldsymbol{θ}_n, \Lambda_n / [\kappa_n(\nu_n – d + 1)])$	Posterior for mean
Predictive $\tilde{\mathbf{y}} \mid \mathbf{y}$	$t_{\nu_n – d + 1}(\boldsymbol{θ}_n, (\kappa_n + 1)\Lambda_n / [\kappa_n(\nu_n – d + 1)])$	Posterior predictive distribution

---

11. Intuitive Summary

• The Normal–Inverse–Wishart (NIW) distribution is the multivariate analogue of the univariate Normal–Inverse–χ² model.
• It provides a conjugate Bayesian framework for the joint estimation of the mean vector and covariance matrix.
• Posterior updating is straightforward—parameters are updated by simple algebraic combinations of prior and data summaries.
• Both marginal and predictive posteriors have closed-form Student-t distributions, making inference and simulation efficient.
• The scaled inverse–Wishart model adds flexibility by separating variances and correlations, improving modeling of complex hierarchical structures.

---

In short:

The multivariate normal model with unknown mean and variance uses the Normal–Inverse–Wishart conjugate prior. It yields analytically tractable posteriors, Student-t marginals, and flexible hierarchical extensions through the scaled inverse–Wishart formulation, forming a cornerstone of Bayesian multivariate modeling.