1. Purpose of Large-Sample (Asymptotic) Theory
The main purpose of large-sample theory in Bayesian inference is to explain why and when the normal approximation works well for posterior distributions.
As the amount of data increases (i.e., $n \to \infty$), the posterior distribution of parameters θ\thetaθ typically becomes sharply concentrated and approximately normal, even if:
- The sample size is only moderately large, and
- The true data-generating distribution is not exactly within the model family $p(y \mid \theta)$.
This is known as the asymptotic normality of the posterior distribution.
2. Mathematical Setting and Notation
Let:
- The observed data be $y_1, y_2, \ldots, y_n$, which are independent samples from a true but possibly unknown distribution $f(y)$.
- The model for the data be a parametric family $p(y \mid \theta)$, with parameter vector $\theta \in \Theta \subseteq \mathbb{R}^d$.
- The prior distribution for $\theta$ be $p(\theta)$.
Even if the true distribution $f(y)$ is not contained in the parametric family $p(y \mid \theta)$, large-sample results can still be derived by introducing a “best-fitting” parameter value $\theta_0$ that minimizes the Kullback–Leibler (KL) divergence between $f(y)$ and $p(y \mid \theta)$:
$\theta_0 = \arg\min_\theta \, \text{KL}\big(f(y)\,\|\,p(y\mid\theta)\big) = \arg\min_\theta \, \int f(y)\log\frac{f(y)}{p(y\mid\theta)}\,dy$
Thus, $\theta_0$ represents the parameter value under which the model $p(y \mid \theta)$ is closest to the true data-generating process.
3. Asymptotic Normality and Consistency
The core asymptotic result is that, under certain regularity conditions,
the posterior distribution becomes approximately normal as the number of observations $n$ grows.
(a) Conditions
- $y_1, \ldots, y_n$ are independent and identically distributed (i.i.d.) from $f(y)$.
- The likelihood function $p(y \mid \theta)$ is smooth and continuous in $\theta$.
- The true (or pseudo-true) parameter $\theta_0$ is not on the boundary of the parameter space.
Under these conditions, the posterior distribution satisfies:
$p(\theta \mid y_1, \ldots, y_n) \approx N\!\left(\theta_0, \, [n J(\theta_0)]^{-1}\right)$
where $J(\theta_0)$ is the Fisher information matrix defined as:
$J(\theta_0) = -E_{f(y)}\!\left[\frac{d^2}{d\theta^2}\log p(y \mid \theta_0)\right]$
(b) Consistency
As $n \to \infty$:
- The posterior mode $\hat{\theta}$ converges to $\theta_0$: $\hat{\theta} \xrightarrow{p} \theta_0$,
- The posterior variance shrinks at rate $1/n$, and
- The posterior mass becomes concentrated in neighborhoods of $\theta_0$, meaning:
- $p(\theta \mid y_1, \ldots, y_n) \text{ collapses to a point mass at } \theta_0.$
This property is called posterior consistency.
If $f(y) = p(y \mid \theta_0)$ for some true parameter $\theta_0$, then the posterior converges to a degenerate distribution at the true parameter.
If not, the posterior still centers around the KL-optimal value $\theta_0$.
4. Derivation Intuition Using Taylor Expansion
Starting from the Taylor expansion of the log posteriorv:
$\log p(\theta \mid y) = \log p(\hat{\theta} \mid y) + \frac{1}{2} (\theta – \hat{\theta})^{T} \left.\frac{d^2}{d\theta^2}\log p(\theta \mid y)\right|_{\theta=\hat{\theta}} (\theta – \hat{\theta}) + \cdots$
The second derivative term (curvature) of the log posterior can be decomposed into two parts:
$\left.\frac{d^2}{d\theta^2}\log p(\theta \mid y)\right|_{\theta=\hat{\theta}} = \left.\frac{d^2}{d\theta^2}\log p(\theta)\right|_{\theta=\hat{\theta}} + \sum_{i=1}^{n} \left.\frac{d^2}{d\theta^2}\log p(y_i \mid \theta)\right|_{\theta=\hat{\theta}}$
- The first term is the curvature contributed by the prior.
- The second term is the curvature contributed by the likelihood.
Taking the expected value of the second term under the true distribution $f(y) = p(y \mid \theta_0)$ gives approximately:
$E\left[\frac{d^2}{d\theta^2}\log p(y_i \mid \theta)\right] \approx -J(\theta_0)$
where $J(\theta_0)$ is the Fisher information.
Therefore, as $n \to \infty$:
$\left.\frac{d^2}{d\theta^2}\log p(\theta \mid y)\right|_{\theta=\hat{\theta}} \approx -n J(\theta_0)$
and hence, $p(\theta \mid y) \approx N\!\left(\hat{\theta}, \, [n J(\hat{\theta})]^{-1}\right).$
This shows that the posterior curvature becomes approximately proportional to the Fisher information as the sample grows.
5. Likelihood Dominating the Prior
A crucial implication of asymptotic theory is that the influence of the prior diminishes as the sample size increases.
Because the log-posterior’s curvature grows linearly with $n$, while the prior term remains fixed, the likelihood term dominates for large $n$:
$log p(\theta \mid y) = \log p(\theta) + \sum_{i=1}^{n}\log p(y_i \mid \theta) \approx \sum_{i=1}^{n}\log p(y_i \mid \theta) \quad (\text{for large } n)$
Thus:
- The posterior mean, mode, and variance depend primarily on the likelihood.
- The prior only matters when $n$ is small or when data are scarce.
In practice, this means that for large datasets, the normal approximation derived purely from the likelihood is usually sufficient.
However, for small samples, an accurately formulated prior remains essential to stabilize inference.
6. Summary of Asymptotic Results
| Concept | Description |
|---|---|
| Asymptotic normality | As $n \to \infty$, $p(\theta \mid y) \approx N(\theta_0, [nJ(\theta_0)]^{-1})$. |
| Consistency | Posterior mode $\hat{\theta} \to \theta_0$; posterior collapses to a point mass. |
| Curvature | Posterior curvature ≈ $nJ(\theta_0)$, the Fisher information times sample size. |
| Likelihood dominance | Prior becomes negligible as $n$ grows; posterior ≈ likelihood-based. |
| Practical implication | Normal approximation becomes increasingly accurate for large data. |
7. Intuitive Interpretation
- As data accumulates, the posterior becomes sharply peaked near the true or KL-optimal parameter value.
- The spread (variance) of the posterior shrinks as $1/n$, and uncertainty decreases accordingly.
- The posterior’s shape becomes increasingly quadratic (normal), regardless of the true data distribution.
- Therefore, the normal approximation is not just a computational trick but a theoretical consequence of Bayesian asymptotics.
8. Practical Implications
- For large $n$:
- The prior $p(\theta)$ is less important.
- Posterior ≈ Likelihood-based normal approximation.
- Standard errors can be derived from Fisher information.
- For small $n$:
- The prior plays a crucial role in stabilizing estimates and avoiding overfitting.
- The normal approximation may fail (skewed or heavy-tailed posteriors).
Thus, large-sample theory not only justifies the normal approximation but also clarifies its limits.
9. Core Takeaways
- Posterior asymptotic normality:
- $p(\theta \mid y_1, \ldots, y_n) \approx N(\theta_0, [nJ(\theta_0)]^{-1})$
- Posterior consistency:
- $\hat{\theta} \to \theta_0, \quad \text{and} \quad \text{Var}(\theta \mid y) \sim O(1/n)$
- Likelihood dominance:
The contribution of the prior becomes negligible compared to the likelihood as $n$ increases. - Practical interpretation:
The normal approximation works because, asymptotically, all smooth posterior distributions behave like quadratic functions around their peak.
In essence:
As sample size grows, the posterior distribution becomes increasingly concentrated and normal-shaped around the true or best-fitting parameter value, with variance determined by the inverse Fisher information.
The prior’s influence fades, and inference becomes almost entirely driven by the likelihood.