1. Overview of Metropolis-Hastings Jumping Rules

Even after reparameterizing a model, there remains an infinite number of possible Metropolis-Hastings (MH) jumping rules (proposal distributions).
For nonconjugate posteriors, MH must be used either alone or inside a Gibbs sampler, making the choice of proposal distribution crucial for efficiency.

There are two major classes of simple jumping rules:

(1) Random-walk proposals

  • Proposals are centered at the current parameter value.
  • Typically use multivariate normal distributions.
  • Efficiency depends strongly on the step-size (scale).

(2) Approximation-based proposals

  • The proposal attempts to mimic the shape of the true posterior (joint or conditional).
  • The MH acceptance step mainly corrects approximation errors.
  • Updating one parameter at a time offers no inherent advantage beyond computational savings.

2. Optimal Random-Walk Proposals for Multivariate Normal Targets

Assume that after proper transformation, the posterior of $\theta = (\theta_1, \ldots, \theta_d)$

is approximately multivariate normal with covariance matrix Σ.

We use a normal random-walk proposal:

$J(\theta^\*|\theta_t) = N(\theta^\*|\theta_t,\, c^2 \Sigma)$

Optimal scale

The most efficient scale is:

$c \approx \frac{2.4}{\sqrt{d}}$

Efficiency

  • Efficiency ≈ 0.3/d compared to independent sampling.
  • A Gibbs sampler with fully independent parameters would give ≈ 1/d.
  • Best choice depends on computational cost per iteration.

3. Optimal Acceptance Rates

For random-walk proposals with shape matching the target posterior:

  • 1 dimension: optimal acceptance ≈ 0.44
  • High dimensions (d > 5): optimal acceptance ≈ 0.23

These are widely used practical guidelines.

4. Adaptive Metropolis Strategy

Using the optimal acceptance rates, the book proposes a two-step adaptive approach:

Phase 1: Initial run

Start with:

  • A Gibbs sampler, or
  • A random-walk MH proposal shaped like an estimate of the target (e.g., inverse Hessian at mode),
  • Scale set to $2.4/\sqrt{d}$​.

Phase 2: Adaptation

After accumulating some samples:

  1. Update the proposal covariance
    Replace the covariance of the jumping distribution with an estimate of the posterior covariance from the simulations.
  2. Adjust the proposal scale
    • If acceptance is too high → increase scale.
    • If acceptance is too low → decrease scale.
      Goal:
    • 0.44 for 1-D updates
    • 0.23 for high-dimensional vector updates

This approach is effective for problems with 1 ≤ d ≤ 50.

5. Dangers of Adaptive Algorithms

If an algorithm continually adapts while running, it may no longer converge to the true posterior.

Reason:

  • Standard MH is based on fixed transition probabilities.
  • If proposals depend on past history of the chain, the Markov property is broken.
  • Example:
    If adaptation speeds up moves in flat regions and slows down moves in steep regions, the simulation will oversample steep regions and undersample flat regions, producing a biased sample.

6. Safe Use: Two-Phase Adaptive Procedure

To avoid convergence errors:

Phase 1 — Adaptive Tuning Phase

  • The algorithm may adjust covariance, scale, etc.
  • Goal: find an efficient jumping rule.
  • Samples from this phase are discarded.

Phase 2 — Fixed Algorithm Phase

  • Freeze the jumping rule (covariance + scale).
  • Continue sampling long enough to reach approximate convergence.
  • Only samples from this fixed phase are retained for inference.

This ensures the final draws are from the correct posterior.

Core Takeaways

  • Optimal MH random-walk proposal for multivariate normal targets uses scale 2.4/√d.
  • Optimal acceptance rate ≈ 0.44 (1-D), 0.23 (high-D).
  • Adaptive MH can improve efficiency but must stop before collecting final samples.
  • Safe workflow: adapt → freeze → sample.