1. Overview
The normal distribution is fundamental to statistical modeling.
Due to the Central Limit Theorem, many real-world data-generating processes can be approximated by a normal likelihood, even if the underlying process is not exactly normal.
When the variance (σ²) is known, we can derive analytical Bayesian results for the mean parameter θ, starting with one observation and then generalizing to multiple data points.
2. Likelihood for One Data Point
Let $y$ be a single scalar observation drawn from a normal distribution:
$p(y \mid θ) = \frac{1}{\sqrt{2\pi σ^2}} \exp\left[-\frac{1}{2σ^2}(y – θ)^2\right]$
Here, $θ$ is the unknown mean and $σ^2$ is the known variance.
3. Conjugate Prior and Posterior
Because the likelihood is an exponential of a quadratic function in θ, the conjugate prior must also be normal.
We parameterize it as:
$p(θ) \propto \exp\left[-\frac{1}{2τ_0^2}(θ – μ_0)^2\right]$
which is:
$θ \sim N(μ_0, τ_0^2)$
where:
- $μ_0$: prior mean
- $τ_0^2$: prior variance (known hyperparameter)
Posterior Distribution
Given $y$, the posterior is also normal:
$p(θ \mid y) \propto \exp\left[-\frac{1}{2}\left(\frac{(y – θ)^2}{σ^2} + \frac{(θ – μ_0)^2}{τ_0^2}\right)\right]$
By expanding and completing the square, we find:
$p(θ \mid y) \propto \exp\left[-\frac{1}{2τ_1^2}(θ – μ_1)^2\right]$
Hence:
$θ \mid y \sim N(μ_1, τ_1^2)$
where
$\frac{1}{τ_1^2} = \frac{1}{τ_0^2} + \frac{1}{σ^2}$
and
$μ_1 = \frac{\frac{μ_0}{τ_0^2} + \frac{y}{σ^2}}{\frac{1}{τ_0^2} + \frac{1}{σ^2}}$
4. Interpretation: Posterior Precision and Mean
- The precision (inverse variance) adds up linearly:
$\text{Posterior precision} = \text{Prior precision} + \text{Data precision}$
That is:
$\frac{1}{τ_1^2} = \frac{1}{τ_0^2} + \frac{1}{σ^2}$
- The posterior mean $μ_1$ is a weighted average of the prior mean and observed data:
$μ_1 = \frac{(1/τ_0^2) μ_0 + (1/σ^2) y}{(1/τ_0^2) + (1/σ^2)}$
Alternative Forms
- As prior adjusted toward the data:
$μ_1 = μ_0 + (y – μ_0) \frac{τ_0^2}{σ^2 + τ_0^2}$
- As data shrunk toward the prior:
$μ_1 = y – (y – μ_0) \frac{σ^2}{σ^2 + τ_0^2}$
Thus, the posterior mean lies between $μ_0$ and $y$, representing a compromise between prior belief and observed evidence.
Limiting Cases
| Condition | Interpretation | Posterior Mean |
|---|---|---|
| $τ_0^2 = 0$ | Prior is infinitely precise | $μ_1 = μ_0$ |
| $σ^2 = 0$ | Data is perfectly precise | $μ_1 = y$ |
| $y = μ_0$ | Data equals prior mean | $μ_1 = μ_0 = y$ |
5. Posterior Predictive Distribution
We now consider a future observation $\tilde{y}$.
The posterior predictive distribution integrates over uncertainty in θ:
$p(\tilde{y} \mid y) = \int p(\tilde{y} \mid θ)p(θ \mid y)\, dθ$
Given $\tilde{y} \mid θ \sim N(θ, σ^2)$ and $θ \mid y \sim N(μ_1, τ_1^2)$, the result of this convolution is also normal:
$\tilde{y} \mid y \sim N(μ_1, σ^2 + τ_1^2)$
Hence:
$E(\tilde{y} \mid y) = μ_1, \quad \operatorname{Var}(\tilde{y} \mid y) = σ^2 + τ_1^2$
The variance combines:
- $σ^2$: predictive variance (data uncertainty)
- $τ_1^2$: posterior uncertainty in θ
6. Extension to Multiple Observations
Now consider $y = (y_1, y_2, \ldots, y_n)$, where all $y_i$ are independent and identically distributed:
$y_i \mid θ \sim N(θ, σ^2)$
The joint likelihood is:
$p(y \mid θ) \propto \exp\left[-\frac{1}{2σ^2} \sum_{i=1}^n (y_i – θ)^2 \right]$
Combining with the prior $N(μ_0, τ_0^2)$, we obtain the posterior:
$θ \mid y \sim N(μ_n, τ_n^2)$
where
$\frac{1}{τ_n^2} = \frac{1}{τ_0^2} + \frac{n}{σ^2}$
and
$μ_n = \frac{\frac{μ_0}{τ_0^2} + \frac{n\bar{y}}{σ^2}}{\frac{1}{τ_0^2} + \frac{n}{σ^2}}$
with $\bar{y} = \frac{1}{n}\sum_{i=1}^n y_i$ being the sample mean.
Thus, $\bar{y}$ is a sufficient statistic for θ in this model.
7. Sequential Updating
If data arrive one at a time, you can update sequentially:
- Use the posterior after the first observation as the new prior for the next.
- After n updates, the posterior is the same as the formula above.
8. Large Sample Approximation
If the prior is weakly informative (large $τ_0^2$) or the data size n is large, the prior influence fades.
As $τ_0^2 \to \infty$ or $n \to \infty$:
$p(θ \mid y) \approx N\left(\bar{y}, \frac{σ^2}{n}\right)$
This is the sampling distribution of the sample mean, confirming that for large n, the Bayesian posterior coincides with the frequentist result.
Key Takeaways
| Concept | Formula / Expression | Interpretation |
|---|---|---|
| Likelihood | $p(y \mid θ) = N(θ, σ^2)$ | Model for data |
| Prior | $θ \sim N(μ_0, τ_0^2)$ | Belief before seeing data |
| Posterior | $θ \mid y \sim N(μ_1, τ_1^2)$ | Updated belief |
| Posterior Mean | $μ_1 = \frac{(1/τ_0^2) μ_0 + (1/σ^2) y}{(1/τ_0^2) + (1/σ^2)}$ | Weighted average of prior and data |
| Posterior Variance | $\frac{1}{τ_1^2} = \frac{1}{τ_0^2} + \frac{1}{σ^2}$ | Additive in precision |
| Predictive Distribution | $\tilde{y} \mid y \sim N(μ_1, σ^2 + τ_1^2)$ | Accounts for model + parameter uncertainty |
| Multiple Observations | $θ \mid y \sim N(μ_n, τ_n^2)$ | Depends only on sample mean $\bar{y}$ |
| Large n Approximation | $θ \mid y \approx N(\bar{y}, σ^2/n)$ | Prior negligible when data dominate |
Summary in words:
The normal model with known variance provides one of the simplest and most elegant examples of Bayesian inference.
- The posterior remains normal due to conjugacy.
- The posterior mean represents a balance between prior belief and observed data.
- The posterior variance reflects combined uncertainty, decreasing as data accumulate.
- The predictive distribution incorporates both data variability (σ²) and parameter uncertainty (τ₁²).
- For large samples, the Bayesian posterior closely approximates the frequentist result $N(\bar{y}, σ^2/n)$.