Normal Data with a Conjugate Prior Distribution

1. Model Setup

We observe $n$ independent data points:

$y_i \mid μ, σ^2 \sim N(μ, σ^2)$

where both $μ$ (mean) and $σ^2$ (variance) are unknown.
Instead of using a noninformative prior, we now assume a conjugate prior that leads to analytically convenient posterior updates.

---

2. The Conjugate Prior Family

The conjugate prior for $(μ, σ^2)$ in the normal model must have the form:

$p(μ, σ^2) = p(μ \mid σ^2)\, p(σ^2)$

where:

$μ \mid σ^2 \sim N(μ_0, \frac{σ^2}{κ_0}), \quad σ^2 \sim \text{Inv-}\chi^2(ν_0, σ_0^2)$

This means:

• $μ$ given $σ^2$ is normal,
• $σ^2$ marginally is scaled inverse-χ².

This pair defines the Normal–Inverse–χ² (N–Inv–χ²) prior family, parameterized by:

$p(μ, σ^2) \propto (σ^2)^{-(ν_0/2 + 1)} \exp\!\left[-\frac{1}{2σ^2}\left(ν_0 σ_0^2 + κ_0 (μ – μ_0)^2\right)\right]$ (3.6)

We denote this joint prior as:

$(μ, σ^2) \sim \text{N-Inv-}\chi^2(μ_0, \frac{σ_0^2}{κ_0}; ν_0, σ_0^2)$

---

3. Interpretation of the Prior Parameters

Parameter	Meaning
$μ_0$	Prior mean (belief about true mean before data)
$κ_0$	Prior strength for $μ$ (like an equivalent sample size)
$ν_0$	Prior degrees of freedom (controls certainty about $σ^2$)
$σ_0^2$​	Prior scale (roughly the expected value of $σ^2$)

---

4. Dependence Between $μ$ and $σ^2$

Because $μ \mid σ^2$ depends on $σ^2$, the joint prior couples them:
if $σ^2$ is large, the prior variance of $μ$ also increases.

This dependence is intuitive:

• If the data-generating process has large measurement variability ($σ^2$ large),
  then uncertainty about the mean $μ$ should also be larger.
• $κ_0$​ can be interpreted as the number of prior equivalent observations.

---

5. Posterior Distribution

Using Bayes’ rule, we multiply the prior by the likelihood:

$p(μ, σ^2 \mid y) \propto p(y \mid μ, σ^2)\, p(μ, σ^2)$

The likelihood (as before) is:

$p(y \mid μ, σ^2) \propto (σ^2)^{-n/2} \exp\!\left[-\frac{1}{2σ^2}((n-1)s^2 + n(\bar{y} – μ)^2)\right]$

Multiplying this with the prior (3.6), we get:

$p(μ, σ^2 \mid y) \propto (σ^2)^{-(ν_0/2 + n/2 + 1)} \exp\!\left[-\frac{1}{2σ^2} \big(ν_0 σ_0^2 + κ_0 (μ – μ_0)^2 + (n-1)s^2 + n(\bar{y} – μ)^2\big)\right]$

After simplification, this posterior belongs to the same family:

$(μ, σ^2) \mid y \sim \text{N-Inv-}\chi^2(μ_n, \frac{σ_n^2}{κ_n}; ν_n, σ_n^2)$

---

6. Posterior Parameter Updates

The updated (posterior) hyperparameters are:

$\begin{aligned} κ_n &= κ_0 + n, \\[6pt] ν_n &= ν_0 + n, \\[6pt] μ_n &= \frac{κ_0 μ_0 + n \bar{y}}{κ_0 + n}, \\[6pt] ν_n σ_n^2 &= ν_0 σ_0^2 + (n-1)s^2 + \frac{κ_0 n}{κ_0 + n} (\bar{y} – μ_0)^2. \end{aligned}$

---

7. Interpretation of Posterior Parameters

• Posterior mean (μₙ):
  Weighted average of prior mean $μ_0$​ and data mean $\bar{y}$​:
  • $μ_n = \frac{κ_0 μ_0 + n \bar{y}}{κ_0 + n}.$ The larger $κ_0$​, the more influence the prior mean has.
• Posterior degrees of freedom (νₙ):
  Equal to $ν_0 + n$, combining prior and data “information.”
• Posterior scale (σₙ²):
  Combines prior scale, sample variance, and disagreement between data and prior mean.

---

8. Conditional Posterior for $μ \mid σ^2, y$

Given $σ^2$, the conditional posterior of $μ$ is:

$μ \mid σ^2, y \sim N\!\left(μ_n, \frac{σ^2}{κ_n}\right)$

Explicitly:

$μ_n = \frac{\frac{κ_0}{σ^2} μ_0 + \frac{n}{σ^2} \bar{y}}{\frac{κ_0}{σ^2} + \frac{n}{σ^2}}, \quad \text{Var}(μ \mid σ^2, y) = \frac{σ^2}{κ_0 + n}$

This matches the known-variance Bayesian updating rule from earlier sections.

---

9. Marginal Posterior for $σ^2 \mid y$

Integrating out $μ$, we get:

$σ^2 \mid y \sim \text{Inv-}\chi^2(ν_n, σ_n^2)$

Thus, the variance posterior depends on both prior information and sample variability.

---

10. Sampling from the Posterior

To simulate from the joint posterior:

1. Draw $σ^2$ from $\text{Inv-}\chi^2(ν_n, σ_n^2)$.
2. Given $σ^2$, draw $μ$ from $N(μ_n, σ^2 / κ_n)$.

This sequential sampling mirrors the hierarchical structure of the model.

---

11. Marginal Posterior for μμμ

Integrating out $σ^2$ yields a Student-t distribution:

$μ \mid y \sim t_{ν_n}\!\left(μ_n, \frac{σ_n}{\sqrt{κ_n}}\right)$

That is:

$p(μ \mid y) \propto \left[1 + \frac{κ_n (μ – μ_n)^2}{ν_n σ_n^2}\right]^{-(ν_n + 1)/2}$

This distribution expresses the posterior uncertainty about the mean after fully accounting for the unknown variance.

---

12. Summary Table

Quantity	Distribution	Parameters	Interpretation
$μ, σ^2$ (prior)	N–Inv–χ²	$μ_0, κ_0, ν_0, σ_0^2$	Prior beliefs
$μ \mid σ^2, y$	Normal	$μ_n, σ^2/κ_n$	Conditional posterior
$σ^2 \mid y$	Inverse-χ²	$ν_n, σ_n^2$	Marginal posterior for variance
$μ \mid y$	Student-t	$/κnν_n, μ_n, σ_n^2/κ_n$	Marginal posterior for mean

---

13. Intuitive Summary

• The conjugate prior makes Bayesian updating analytically tractable.
• Posterior parameters blend prior and data in proportion to their relative precision.
• The posterior of $μ$ remains t-distributed, just like in the noninformative case—but with updated degrees of freedom and precision reflecting the prior.
• The model links the uncertainty about $μ$ to the uncertainty about $σ^2$, a natural and realistic dependency.

---

14. Key Insight

The Normal–Inverse–χ² prior family provides a coherent and interpretable Bayesian framework for joint inference on both the mean and variance of normal data.
It generalizes the noninformative prior by introducing flexible parameters that represent:

• how much prior information we have ($κ_0, ν_0$​),
• what we believe about the mean ($μ_0$​),
• and how variable the system is ($σ_0^2$​).

In short:

Noninformative prior = special case of the conjugate prior with $κ_0, ν_0 \to 0$.