Overview
The Gibbs sampler — also known as alternating conditional sampling — is a Markov Chain Monte Carlo (MCMC) algorithm widely used for generating samples from high-dimensional posterior distributions when direct sampling from the joint distribution is difficult.
It works by iteratively sampling each parameter (or block of parameters) from its conditional distribution, given the current values of all other parameters.
1. General structure
Suppose the full parameter vector is divided into $d$ components:
$θ = (θ_1, θ_2, …, θ_d)$
Each iteration $t$ of the Gibbs sampler consists of $d$ steps:
At each step, for
$j = 1, …, d$, $θ_j^{(t)} \sim p(θ_j \mid θ_1^{(t)}, …, θ_{j-1}^{(t)}, θ_{j+1}^{(t-1)}, …, θ_d^{(t-1)}, y)$
That is, the sampler updates each subvector $θ_j$ using its conditional posterior distribution, given the most recent values of the other components.
- The already-updated components use their iteration $t$ values.
- The not-yet-updated components use their previous iteration $t – 1$ values.
By cycling through these conditional distributions repeatedly, the sampler produces a sequence of dependent draws that converge to the joint posterior distribution $p(θ \mid y)$.
2. Why Gibbs sampling works
Many hierarchical or standard statistical models are built from conditional distributions, often chosen to be conjugate, making them easy to sample directly.
Thus, the Gibbs sampler can exploit these analytic conditional forms without requiring numerical optimization or rejection steps.
3. Example — Bivariate normal distribution
Consider one observation $(y_1, y_2)$ from a bivariate normal population with:
- Unknown mean vector $θ = (θ_1, θ_2)$
- Known covariance matrix:
$\Sigma = \begin{pmatrix} 1 & ρ \\ ρ & 1 \end{pmatrix}Σ$
With a uniform prior on $θ$, the posterior distribution is:
$θ \mid y \sim N \left( \begin{pmatrix} y_1 \\ y_2 \end{pmatrix}, \Sigma \right)$
Although we could draw directly from this joint normal posterior, we use it to illustrate Gibbs sampling.
Conditional posterior distributions
From multivariate normal properties, the conditional distributions are:
$θ_1 \mid θ_2, y \sim N(y_1 + ρ(θ_2 – y_2), 1 – ρ^2)$
$θ_2 \mid θ_1, y \sim N(y_2 + ρ(θ_1 – y_1), 1 – ρ^2)$
Sampling procedure
- Initialize $θ_1^{(0)}, θ_2^{(0)}$ (e.g., random or overdispersed values).
- For each iteration $t$:
- Draw $θ_1^{(t)}$ from $p(θ_1 \mid θ_2^{(t-1)}, y)$
- Draw $θ_2^{(t)}$ from $p(θ_2 \mid θ_1^{(t)}, y)$
- Repeat until convergence.
4. Illustration
For example, let:
- $ρ = 0.8$
- Data $(y_1, y_2) = (0, 0)$
- Four chains start at overdispersed points: $(±2.5, ±2.5)$
Results:
- Early iterations (e.g., first 10): show alternating updates, gradually moving toward the center.
- After ~500 iterations: the chains have mixed and converged to the target posterior.
- After convergence: the second halves of the sequences provide correlated samples from $p(θ_1, θ_2 \mid y)$.
Key Takeaways
| Concept | Explanation |
|---|---|
| Purpose | Generate samples from a multivariate posterior by iteratively sampling from conditional distributions. |
| Process | Each variable is updated given the most recent values of all others. |
| When useful | When direct joint sampling is difficult but conditionals are easy to simulate. |
| Advantages | Exploits conjugate structures; simple to implement. |
| Limitations | Can converge slowly when parameters are strongly correlated (high dependence). |
| Example | Bivariate normal model shows gradual convergence to the true joint posterior. |
Summary:
The Gibbs sampler constructs a Markov chain by repeatedly sampling each parameter (or block) from its conditional posterior distribution given the others, producing dependent draws that eventually approximate the full joint posterior $p(θ \mid y)$.