1. Overview

This is about approximate Bayesian methods that are not standard MCMC:

  • INLA (Integrated Nested Laplace Approximation)
    – For certain hierarchical models with a small number of hyperparameters.
  • CCD (Central Composite Design integration)
    – A clever deterministic way to approximate integrals around the posterior mode using a moderate number of points.
  • ABC (Approximate Bayesian Computation)
    – Likelihood-free methods that accept or reject parameter draws based on simulated data being “close” to the observed data.
  • Substitution likelihood
    – Replace the “hard” likelihood with a simpler, approximate one (e.g., rank-based or quantile-based).

2. INLA – Integrated Nested Laplace Approximation

2.1 Basic idea

INLA is designed for hierarchical models, especially models where:

  • You can split parameters into:
    • A large block:
      • $\gamma \quad (\text{often latent variables, e.g. random effects, spatial field})$
    • A small block of hyperparameters:
      • $\phi \quad (\text{e.g. variance parameters, range parameters, etc.})$
  • The conditional posterior of γ given φ and data y, $p(\gamma \mid \phi, y)$ can be well approximated by a multivariate normal.

The key decomposition:
We want

$p(\gamma, \phi \mid y)$.

Write:

$p(\gamma, \phi \mid y) = p(\gamma \mid \phi, y)\, p(\phi \mid y)$

But often:

  • γ is high-dimensional (e.g. thousands of latent variables),
  • φ is low-dimensional (maybe 2–10 parameters).

So INLA tries to:

  1. Approximate $p(\gamma \mid \phi, y)$ as Gaussian:
    • $p(\gamma \mid \phi, y) \approx p_{\text{approx}}(\gamma \mid \phi, y) = N(\gamma \mid m(\phi), \Sigma(\phi))$
  2. Use the ratio trick to derive an approximation to $p(\phi \mid y)$:
    • $p_{\text{approx}}(\phi \mid y) = \frac{p(\gamma, \phi \mid y)}{p_{\text{approx}}(\gamma \mid \phi, y)} \Bigg|_{\gamma = \gamma^\*(\phi)}$
    • where $\gamma^\*(\phi)$ is usually chosen as the mode or mean of the approximate conditional distribution.
  3. Once we have $p_{\text{approx}}(\phi \mid y)$ on a grid of φ values, we can:
    • Normalize it,
    • Approximate integrals with respect to φ by finite sums over the grid.

Finally, to get marginals for the components of γ, say γᵢ:

$p(\gamma_i \mid y) = \int p(\gamma_i \mid \phi, y) \, p(\phi \mid y)\, d\phi \approx \sum_{\phi \in \text{grid}} p_{\text{approx}}(\gamma_i \mid \phi, y)\, p_{\text{approx}}(\phi \mid y)\, \Delta \phi$

This is the “integrated nested Laplace” idea:

  • Nested: there’s an inner Laplace approximation for $p(\gamma \mid \phi, y)$,
  • Integrated: then we integrate over φ numerically.

2.2 When INLA works well

INLA is powerful when:

  • The dimension of φ is small, so that:
    • Building a grid over φ is feasible,
    • Evaluating the approximate marginal $p_{\text{approx}}(\phi \mid y)$ at each grid point is manageable.
  • The conditional distribution $p(\gamma \mid \phi, y)$ is close to Gaussian, which happens naturally when γ has a joint normal prior and you’re in certain hierarchical / latent Gaussian models.

This is why INLA is very popular for:

  • Spatial models,
  • Time series of random effects,
  • Generalized linear mixed models where the latent layer is Gaussian.

3. CCD – Central Composite Design integration

3.1 Motivation

Suppose:

  • You have a posterior over φ (or θ),
  • You want to approximate integrals like:
    • $\int h(\phi)\, p(\phi \mid y)\, d\phi$
  • The posterior is concentrated near a mode, and its shape near the mode matters,
  • But:
    • Full grid integration is too expensive (curse of dimensionality),
    • You want something more accurate than a single Laplace approximation.

Central Composite Design (CCD) is a deterministic integration scheme that:

  • Places evaluation points in a smart way around the mode,
  • Uses special weights to approximate integrals,
  • Needs far fewer points than a full grid, but more points than just the mode.

3.2 How CCD works conceptually

For a parameter vector φ of dimension d:

  • CCD places points:
    • At the mode,
    • At points along each coordinate axis (± directions),
    • At certain combinations of axes (fractional factorial design),
    • At some “star points” to capture curvature.

This layout allows:

  • Estimating the local curvature of the log-posterior around the mode,
  • Approximating integrals as a weighted finite sum over these points.
    • For d = 5, CCD uses 27 integration points,
    • For d = 15, CCD uses 287 points.

Compare that to a full grid with even, say, 3 or 5 points per dimension; that would explode combinatorially. CCD avoids this exponential blow-up while still capturing:

  • Main effects,
  • Some interactions,
  • Overall curvature.

3.3 Accuracy level

Roughly, CCD’s accuracy lies:

  • Better than a pure modal (Laplace) approximation,
  • Worse than a full fine grid integration or a very large Monte Carlo sample,

but with a fixed, moderate number of posterior evaluations. So it’s useful when:

  • Evaluating $p(\phi \mid y)$ is fairly expensive,
  • Dimensionality is “moderate” (not tiny, not huge),
  • You want a deterministic method with controlled number of evaluations.

4. ABC – Approximate Bayesian Computation

4.1 Big picture

ABC is for situations where:

  • It’s easy to simulate data from the model $y^{\text{rep}} \sim p(y \mid \theta)$,
  • But it’s hard or impossible to compute the likelihood $p(y \mid \theta)$ analytically.

So ABC approximates the posterior without directly using the likelihood. Classic high-level idea:

  • Draw θ from some distribution (often the prior),
  • Simulate $y_{\text{rep}}$ from the model given θ,
  • Compare $y_{\text{rep}}$ to actual data y,
  • Accept θ if $y_{\text{rep}}$ is “close” to y, otherwise reject.

The accepted θ’s (if the tolerance is small enough) approximate draws from something like the posterior.

4.2 Basic rejection ABC algorithm

  1. Draw $\theta \sim p(\theta).$
  2. Simulate replicated data $y^{\text{rep}} \sim p(y^{\text{rep}} \mid \theta).$
  3. Compute a discrepancy measure:
    • $d(y^{\text{rep}}, y)$
    • where:
      • $d = 0$ if $y_{\text{rep}}$ = y exactly,
      • Larger d means “more different.”
  4. Accept θ if $d(y^{\text{rep}}, y) < \epsilon$, for some tolerance ε; otherwise reject.

If you repeat this many times, the accepted θ’s are drawn with probability roughly proportional to:

$\Pr(d(y^{\text{rep}}, y) < \epsilon \mid \theta) \approx \text{“likelihood” of θ when using this distance}$

Thus, prior × this approximate likelihood ≈ approximate posterior.

4.3 Three main challenges in ABC

  1. Choosing the discrepancy function d(·,·)
    • Ideally, d should measure distance between summary statistics that are (approximately) sufficient for θ.
    • If you use the raw data directly:
      • You may be demanding too strong a match,
      • You reject almost everything for complex data.
    • If you use irrelevant summaries, the accepted θ’s won’t be well constrained.
    So designing good summaries / discrepancy is critical.
  2. Choosing the tolerance ε
    • If ε is too large:
      • You accept many θ’s,
      • But the posterior is too close to the prior (weak information).
    • If ε is too small:
      • You accept almost nothing,
      • The algorithm becomes computationally infeasible.
  3. Efficiency when the prior is broad
    • If p(θ) is very diffuse, most θ’s generate $y_{\text{rep}}$ that is very far from y,
    • So the acceptance rate can be tiny,
    • You waste a lot of computation.

4.4 Improvements on basic ABC

To address these problems:

  • Better proposals:
    • Instead of sampling θ from the prior, sample from a proposal that is closer to the posterior,
    • Then use importance sampling weights to correct.
  • ABC + MCMC:
    • Use an MCMC algorithm where the likelihood step is replaced with the ABC accept/reject comparison,
    • This can move around in regions of θ space where the discrepancy is small.
  • Sequential Monte Carlo (SMC-ABC):
    • Start with a large ε (high acceptance), then gradually decrease ε,
    • Reweight and move particles toward higher posterior density regions.

ABC is fundamentally based on simulation + acceptance based on closeness, and then notes that combining ABC with importance sampling or MCMC can greatly improve efficiency.

5. Substitution likelihood and related ideas

This is a related but slightly different idea from ABC.

Instead of avoiding the likelihood completely, we replace the true likelihood with a simpler “almost-likelihood” that:

  • Is easier to compute,
  • Or focuses on particular aspects of the data.

Examples:

  • Rank likelihood:
    • Use only the ranks of the data, not their exact values,
    • So the approximate likelihood is:
      • $L_{\text{rank}}(\theta) \propto \Pr(\text{observed ranks} \mid \theta)$
  • Quantile-based likelihood:
    • Use only some quantiles of the data,
    • Ignore the details between those quantiles,
    • The “likelihood” becomes something like:
      • $L_{\text{quantile}}(\theta) \propto \Pr(\text{observed quantiles} \mid \theta)$

Then, in Bayes’ rule, we plug in this substitute likelihood in place of the true one:

$p_{\text{sub}}(\theta \mid y) \propto p(\theta) \times L_{\text{substitute}}(\theta)$

Why might we do this?

  • Sometimes we want to apply a copula model or some model for dependence structure, but:
    • The assumed marginal distributions do not fit the data well,
    • Or specifying the full marginals is hard.
  • Using a rank or quantile-based likelihood focuses on order or shape but not exact marginal densities, which can be easier to handle or more robust.

This is a computational approximation in the sense that it lets us use a known joint model (copula) in contexts where the exact marginal likelihood would be ugly or wrong.

6. Summary

  • INLA:
    • Partition parameters into many γ and few φ.
    • Use Gaussian (Laplace) approximation for $p(\gamma \mid \phi, y)$.
    • Build a grid over low-dimensional φ, compute approximate $p(\phi \mid y)$, then integrate over φ to get marginals for γ.
    • Very effective for latent Gaussian hierarchical models with few hyperparameters.
  • CCD (Central Composite Design):
    • A deterministic integration method using carefully chosen points around the mode.
    • Uses far fewer points than a full grid but captures curvature.
    • Accuracy between simple Laplace and full grid / large Monte Carlo.
  • ABC (Approximate Bayesian Computation):
    • Likelihood-free:
      • draw θ, simulate $y^{\text{rep}}$, accept θ if $d(y^{\text{rep}}, y) < \epsilon$.
    • Needs good discrepancy d(·,·), good ε, and usually smarter proposals or MCMC/SMC to be efficient.
    • Powerful when you can simulate from the model but cannot compute $p(y \mid \theta)$.
  • Substitution likelihood:
    • Replace the true likelihood with a simpler “almost-likelihood” (rank likelihood, quantile likelihood, etc.),
    • Then apply Bayes’ rule with that substitute.
    • Useful for copula models or robust modeling where full marginals don’t fit or are hard to specify.