Overview
The Gibbs sampler and the Metropolis(-Hastings) algorithms are two foundational Markov Chain Monte Carlo (MCMC) methods.
They can be combined flexibly to handle complex Bayesian models that mix conditionally conjugate and non-conjugate components.
1. When to Use Gibbs vs. Metropolis
| Algorithm | When to Use | How It Works |
|---|---|---|
| Gibbs Sampler | For conditionally conjugate models, where we can sample directly from each conditional posterior. | Updates each parameter (or block) using its exact conditional posterior distribution. |
| Metropolis Algorithm | For non-conjugate models, where conditional posteriors do not have closed forms. | Proposes a candidate value and accepts/rejects it with probability based on the posterior density ratio. |
Example
- Gibbs Sampler: Suitable for normal–normal hierarchical models.
- Metropolis Algorithm: Suitable for models like two-parameter logistic regression (the bioassay example).
- It can be run either:
- In vector form, proposing simultaneous updates to both parameters (α, β), or
- In a Gibbs-like structure, alternating one-dimensional Metropolis updates for α and β separately.
- The proposal distribution in Metropolis steps must usually be tuned to achieve a good acceptance rate.
- It can be run either:
2. Hybrid Gibbs–Metropolis Approach
In many practical Bayesian models:
- Some conditional posteriors can be sampled directly (conjugate forms).
- Others cannot (non-conjugate forms).
In such cases:
- Update parameters one at a time:
- Use Gibbs sampling for the tractable ones.
- Use Metropolis updates for the intractable ones.
- Or, update blocks of parameters together, each block using either a Gibbs step or a Metropolis step.
This hybrid method allows MCMC sampling even in models that are only partly conjugate.
3. Dealing with Correlated Parameters
A common issue in conditional sampling algorithms (Gibbs or hybrid):
- When parameters are highly correlated, the Markov chain moves slowly and convergence is poor.
Solutions:
- Reparameterization: Transform the parameters to reduce correlation.
- Advanced algorithms: Use more efficient methods such as Hamiltonian Monte Carlo or adaptive samplers.
4. Gibbs Sampler as a Special Case of Metropolis–Hastings
The Gibbs sampler can be understood as a special case of the Metropolis–Hastings (MH) algorithm where every proposed move is always accepted.
Setup
- Divide θ into $d$ components: $θ = (θ_1, …, θ_d)$.
- Each iteration $t$ has ddd steps; in step $j$, update $θ_j$ given all others $θ_{\to j}$.
The Gibbs jumping distribution is defined as:
$J^{\text{Gibbs}}_{j,t}(θ^* \mid θ^{(t-1)}) = \begin{cases} p(θ^*_j \mid θ^{(t-1)}_{\to j}, y), & \text{if } θ^*_{\to j} = θ^{(t-1)}_{\to j} \\ 0, & \text{otherwise.} \end{cases}$
Acceptance Ratio
Plugging into the MH ratio:
$r = \frac{p(θ^* \mid y) / J^{\text{Gibbs}}_{j,t}(θ^* \mid θ^{(t-1)})} {p(θ^{(t-1)} \mid y) / J^{\text{Gibbs}}_{j,t}(θ^{(t-1)} \mid θ^*)}$
This simplifies to $r = 1$, meaning every Gibbs move is accepted.
Therefore, the Gibbs sampler is simply the Metropolis–Hastings algorithm with 100% acceptance rate.
Note
- One full iteration usually means updating all $d$ components.
- However, Gibbs sampling still works as long as each component is updated periodically, not necessarily every single iteration.
5. Gibbs Sampler with Approximate Conditionals
Sometimes, even the conditional posteriors $p(θ_j \mid θ_{\to j}, y)$ are not directly sampleable.
In such cases, we can:
- Construct an approximation $g(θ_j \mid θ_{\to j})$ that is easier to sample from.
- Use the Metropolis–Hastings correction to adjust for the approximation.
Modified jumping rule
$J_{j,t}(θ^* \mid θ^{(t-1)}) = \begin{cases} g(θ^*_j \mid θ^{(t-1)}_{\to j}), & \text{if } θ^*_{\to j} = θ^{(t-1)}_{\to j} \\ 0, & \text{otherwise.} \end{cases}$
Then compute the MH ratio $r$ and accept or reject accordingly.
This approach — sometimes called Metropolis-within-Gibbs — allows sampling when conditional distributions are only approximately known or too complex for direct sampling.
Key Takeaways
| Concept | Explanation |
|---|---|
| Combination | Gibbs and Metropolis steps can be mixed within the same model to handle both conjugate and non-conjugate parts. |
| Flexibility | Each parameter (or block) can be updated using whichever method is appropriate. |
| Efficiency issue | Slow mixing can occur when parameters are highly correlated; reparameterization or advanced algorithms can fix this. |
| Gibbs as MH special case | Gibbs sampling = MH where all proposals are always accepted. |
| Approximation-based Gibbs | Use approximate conditional distributions with MH correction when exact ones are unavailable. |
Summary:
The Gibbs and Metropolis algorithms can be combined flexibly to sample from complex posterior distributions; Gibbs is preferred when conditional posteriors are tractable, Metropolis for non-conjugate cases, and the two can be integrated—sometimes with approximations—to create efficient hybrid MCMC methods for general Bayesian inference.