Posterior inference and computation

Robust models or sensitivity-analysis models often expand a simpler baseline model by adding a hyperparameter that controls tail behavior or dispersion. Posterior computation for such models can proceed in two ways:

1. Full Markov chain simulation under the expanded model, often using a mixture representation.
2. Approximate computation using draws already obtained from the original posterior distribution.

Both approaches aim to obtain inference for the expanded posterior distribution$$p(\theta \mid \phi, y) \propto p(\theta \mid \phi)\, p(y \mid \theta, \phi),$$

where $\theta$ is the parameter vector from the original model and $\phi$ is the robustness/sensitivity parameter (for example, the degrees of freedom $\nu$ in a $t$ model).

---

1. Notation for Robust Models

Let:

• $p_0(\theta \mid y)$: posterior distribution from the original model (already computed).
• $\phi$: hyperparameter defining the expanded or robust model.
• Expanded posterior of interest: $$p(\theta \mid \phi, y) \propto p(\theta \mid \phi)\, p(y \mid \theta, \phi).$$

The robust component may enter through:

• Parameter distribution $p(\theta \mid \phi)$, or
• Data distribution $p(y \mid \theta, \phi)$.

A prior must specify the joint distribution $p(\theta, \phi)$, since the relationship between $\theta$ and $\phi$ determines how robustness is expressed.

---

2. Gibbs Sampling via Mixture Representations

Many robust models, including the $t$, negative binomial, and beta-binomial, admit a mixture representation involving additional latent scale parameters:

• For the $t_\nu(\mu, \sigma^2)$ model, the representation is: $$y_i \mid V_i \rightarrow N(\mu, V_i), \qquad V_i \rightarrow \text{Inv-}\chi^2(\nu, \sigma^2).$$

Sampling proceeds by Gibbs updates for:

• $\theta = (\mu, \sigma^2)$
• Latent scales $V_1, \dots, V_n$

If $\nu$ itself is unknown, an additional step must draw from the conditional distribution of $\nu$. This step typically lacks a closed form and requires a Metropolis update.

A complication:

• Posterior densities under robust models may be multimodal.
• Each mode often corresponds to a different observation occupying the tail of the distribution.
• Proper exploration may require strategies such as simulated tempering or careful initialization.

---

3. Posterior Predictive Simulation for Sensitivity Analysis

To simulate future data $\tilde{y}$​ under a robust model:

1. Draw $\theta \sim p(\theta \mid \phi, y)$.
2. Draw $\tilde{y} \sim p(\tilde{y} \mid \theta, \phi)$.

For mixture models:

• Draw the latent scale first: $$V \sim \text{Inv-}\chi^2(\nu, \sigma^2),$$
• Then simulate: $$\tilde{y} \sim N(\mu, V).$$

This two-step structure reflects the model’s long-tailed behavior.

---

4. Computing the Marginal Posterior of the Robustness Parameter $\phi$ via Importance Weighting

If simulation draws $\theta^s$ have already been obtained from the original posterior $p_0(\theta \mid y)$, then the marginal posterior of $\phi$ can be approximated without running a new MCMC chain.

Using the identity:$$p(\phi \mid y) \propto p(\phi) \int \frac{p(\theta \mid \phi)\, p(y \mid \theta, \phi)} {p_0(\theta)\, p_0(y \mid \theta)} \, p_0(\theta \mid y)\, d\theta,$$

the expression can be approximated by averaging importance ratios:$$p(\phi \mid y) \approx p(\phi)\, \frac{1}{S} \sum_{s=1}^S \frac{p(\theta^s \mid \phi)\, p(y \mid \theta^s, \phi)} {p_0(\theta^s)\, p_0(y \mid \theta^s)}.$$

The proportionality constant does not matter when the goal is to compare relative posterior density values across a grid of $\phi$.

---

5. Approximating the Robust Posterior $p(\theta \mid \phi, y)$ via Importance Resampling

When the goal is to approximate the full posterior under a robust model for many values of $\phi$, importance resampling avoids running a new MCMC chain for each value.

Procedure:

1. Begin with a large set of original posterior draws $$\theta^1, \dots, \theta^S \sim p_0(\theta \mid y) \qquad (S \approx 5000).$$
2. For each fixed value of $\phi$, compute importance ratios: $$w(\theta) = \frac{p(\theta \mid \phi)\, p(y \mid \theta, \phi)} {p_0(\theta)\, p_0(y \mid \theta)}.$$
3. Draw a smaller subsample (e.g., $k = 500$) without replacement,
   with probability proportional to $w(\theta^s)$.

This collection is an approximate sample from$$p(\theta \mid \phi, y).$$

A separate subsample must be drawn for each value of $\phi$,
but the same original draws can be reused.

Indicators of sensitivity

If importance weights vary wildly:

• A handful of draws dominate the weighted distribution.
• Approximation becomes unstable.
• Sensitivity to $\phi$ is high.
• Full MCMC under the robust model becomes necessary.