Using Gibbs and Metropolis as building blocks

Overview

The Gibbs sampler and the Metropolis(-Hastings) algorithms are two foundational Markov Chain Monte Carlo (MCMC) methods.
They can be combined flexibly to handle complex Bayesian models that mix conditionally conjugate and non-conjugate components.

---

1. When to Use Gibbs vs. Metropolis

Algorithm	When to Use	How It Works
Gibbs Sampler	For conditionally conjugate models, where we can sample directly from each conditional posterior.	Updates each parameter (or block) using its exact conditional posterior distribution.
Metropolis Algorithm	For non-conjugate models, where conditional posteriors do not have closed forms.	Proposes a candidate value and accepts/rejects it with probability based on the posterior density ratio.

Example

• Gibbs Sampler: Suitable for normal–normal hierarchical models.
• Metropolis Algorithm: Suitable for models like two-parameter logistic regression (the bioassay example).
  • It can be run either:
    • In vector form, proposing simultaneous updates to both parameters (α, β), or
    • In a Gibbs-like structure, alternating one-dimensional Metropolis updates for α and β separately.
  • The proposal distribution in Metropolis steps must usually be tuned to achieve a good acceptance rate.

---

2. Hybrid Gibbs–Metropolis Approach

In many practical Bayesian models:

• Some conditional posteriors can be sampled directly (conjugate forms).
• Others cannot (non-conjugate forms).

In such cases:

• Update parameters one at a time:
  • Use Gibbs sampling for the tractable ones.
  • Use Metropolis updates for the intractable ones.
• Or, update blocks of parameters together, each block using either a Gibbs step or a Metropolis step.

This hybrid method allows MCMC sampling even in models that are only partly conjugate.

---

3. Dealing with Correlated Parameters

A common issue in conditional sampling algorithms (Gibbs or hybrid):

• When parameters are highly correlated, the Markov chain moves slowly and convergence is poor.

Solutions:

• Reparameterization: Transform the parameters to reduce correlation.
• Advanced algorithms: Use more efficient methods such as Hamiltonian Monte Carlo or adaptive samplers.

---

4. Gibbs Sampler as a Special Case of Metropolis–Hastings

The Gibbs sampler can be understood as a special case of the Metropolis–Hastings (MH) algorithm where every proposed move is always accepted.

Setup

• Divide θ into $d$ components: $θ = (θ_1, …, θ_d)$.
• Each iteration $t$ has ddd steps; in step $j$, update $θ_j$​ given all others $θ_{\to j}$​.

The Gibbs jumping distribution is defined as:

$J^{\text{Gibbs}}_{j,t}(θ^* \mid θ^{(t-1)}) = \begin{cases} p(θ^*_j \mid θ^{(t-1)}_{\to j}, y), & \text{if } θ^*_{\to j} = θ^{(t-1)}_{\to j} \\ 0, & \text{otherwise.} \end{cases}$

Acceptance Ratio

Plugging into the MH ratio:

$r = \frac{p(θ^* \mid y) / J^{\text{Gibbs}}_{j,t}(θ^* \mid θ^{(t-1)})} {p(θ^{(t-1)} \mid y) / J^{\text{Gibbs}}_{j,t}(θ^{(t-1)} \mid θ^*)}$

This simplifies to $r = 1$, meaning every Gibbs move is accepted.
Therefore, the Gibbs sampler is simply the Metropolis–Hastings algorithm with 100% acceptance rate.

Note

• One full iteration usually means updating all $d$ components.
• However, Gibbs sampling still works as long as each component is updated periodically, not necessarily every single iteration.

---

5. Gibbs Sampler with Approximate Conditionals

Sometimes, even the conditional posteriors $p(θ_j \mid θ_{\to j}, y)$ are not directly sampleable.
In such cases, we can:

1. Construct an approximation $g(θ_j \mid θ_{\to j})$ that is easier to sample from.
2. Use the Metropolis–Hastings correction to adjust for the approximation.

Modified jumping rule

$J_{j,t}(θ^* \mid θ^{(t-1)}) = \begin{cases} g(θ^*_j \mid θ^{(t-1)}_{\to j}), & \text{if } θ^*_{\to j} = θ^{(t-1)}_{\to j} \\ 0, & \text{otherwise.} \end{cases}$

Then compute the MH ratio $r$ and accept or reject accordingly.

This approach — sometimes called Metropolis-within-Gibbs — allows sampling when conditional distributions are only approximately known or too complex for direct sampling.

---

Key Takeaways

Concept	Explanation
Combination	Gibbs and Metropolis steps can be mixed within the same model to handle both conjugate and non-conjugate parts.
Flexibility	Each parameter (or block) can be updated using whichever method is appropriate.
Efficiency issue	Slow mixing can occur when parameters are highly correlated; reparameterization or advanced algorithms can fix this.
Gibbs as MH special case	Gibbs sampling = MH where all proposals are always accepted.
Approximation-based Gibbs	Use approximate conditional distributions with MH correction when exact ones are unavailable.

---

Summary:

The Gibbs and Metropolis algorithms can be combined flexibly to sample from complex posterior distributions; Gibbs is preferred when conditional posteriors are tractable, Metropolis for non-conjugate cases, and the two can be integrated—sometimes with approximations—to create efficient hybrid MCMC methods for general Bayesian inference.