1. Why sensitivity analysis is necessary

The posterior distribution of model parameters θ can either overestimate or underestimate the true uncertainty.

  • Overestimation occurs because models rarely include all of our substantive prior knowledge.
    → That’s why posterior results must be checked against real-world understanding.
  • Underestimation happens for two reasons:
    1. The assumed model is almost certainly wrong, so model checking against data is essential.
    2. Other reasonable models could fit the same data equally well — this motivates sensitivity analysis.

Hence, sensitivity analysis evaluates how inferences about θ change when plausible alternative models are considered.
Model checking and sensitivity analysis go together — we only test models that make sense substantively and fit the data reasonably well.

2. Core method of sensitivity analysis

The basic method is to fit several alternative probability models to the same dataset and see how posterior results differ.

Key points:

  • Replace improper priors with proper priors that represent real knowledge to avoid unrealistic sensitivity.
  • Different questions react differently to model changes:
    • Medians of posterior distributions are usually less sensitive than means or extreme quantiles.
    • Predictions close to the observed range (interpolation) are more stable than predictions far away (extrapolation).
  • One can use robust models (e.g., t instead of Normal) to reduce the influence of outliers or unusual data points.
    Such robust models require more computation but provide stability.

3. Reasons to expand a model

Model expansion often means adding parameters or generalizing the structure.
Four main motivations:

  1. Model–data mismatch:
    When the model fails to fit data or prior knowledge, add parameters or structure to improve the fit.
  2. Weak or questionable assumptions:
    For instance, replace a Normal assumption with a t-distribution to allow heavier tails.
  3. Bridging two competing models:
    Combine $p_1(y, \theta)$ and $p_2(y, \theta)$ into one larger model with a continuous parameterization that includes both as special cases.
    Example:
    The hierarchical model for SAT coaching generalizes
    • complete pooling ($\tau = 0$) and
    • no pooling ($\tau = \infty$)
      into one continuous framework with $\tau$ as a hyperparameter.
  4. Including new data or data structures:
    • Incorporating new experiments into a hierarchical model.
    • Expanding $p(y|x)$ to a joint $p(x, y)$ model to handle missing values in $x$.

4. Mathematical form of model expansion

A simpler model $p(y, \theta)$ is replaced or embedded in a broader model $p(y, \theta, \phi)$,
or, more generally, $p(y, y^*, \theta, \phi)$ — where $y^*$ represents new or additional data.

Joint posterior under the expanded model:

$p(\theta, \phi \mid y, y^*) \propto p(\phi)\, p(\theta \mid \phi)\, p(y, y^* \mid \theta, \phi)$

Marginal posterior of the new parameters:

$p(\phi \mid y, y^*) \propto p(\phi) \int p(\theta \mid \phi)\, p(y, y^* \mid \theta, \phi)\, d\theta$

Model expansion thus requires:

  • A conditional prior $p(\theta \mid \phi)$
  • A hyperprior $p(\phi)$
    Both must be chosen carefully, especially when prior information is vague.

Examples:

  • Modeling unequal variances → combining unweighted and weighted regressions.
  • Extending the normal model to a t-model by adding degrees of freedom as a new hyperparameter.
  • Building hierarchical mixture models for complex experimental data.

5. Accounting for model choice in data analysis

Usually, the “final model” arises after much exploration of the same dataset — which introduces risks similar to multiple comparisons or overfitting.

  • Hierarchical modeling helps by shrinking related estimates together (e.g., $\theta_j$’s toward $\mu$ in the 8-schools model), which automatically controls exaggerated differences.
  • However, analyzing the same data repeatedly to “find strong results” remains risky and not fully solved by Bayesian modeling.

6. Selecting predictors and combining information

In regression analysis:

  • Many plausible ways to specify a model can lead to very different conclusions.
  • Predictor choice and transformation decisions are forms of model specification.

Preferred strategy:

  • Include as many predictors as reasonable,
  • Group and scale them (e.g., using an ANOVA-style hierarchical structure)
    so that effects are partially pooled instead of being strictly “in” or “out.”

Bayesian model averaging can also help, but continuous hierarchical structures are usually preferred.

In causal inference, variable selection and model structure must reflect substantive causal relationships (e.g., principal stratification methods).

7. Alternative model formulations

Adding parameters can make a model more realistic:

  • Estimating variance instead of fixing it.
  • Using t-distribution instead of Normal to handle heavy tails.

However, model expansion should balance accuracy and convenience — endlessly adding complexity isn’t always helpful.
Predictive model checks help identify when extra complexity is truly needed.

As computational methods for hierarchical modeling become more standard, more flexible models will likely become the default.

8. Practical advice for model checking and expansion

There is no universal rule for model choice — it requires scientific judgment.
However, a systematic workflow is advised:

  1. Examine posterior distributions of key parameters and predictions.
  2. Compare predictions to both data and substantive knowledge.
  3. Identify discrepancies → suggest expansions (e.g., more informative prior, added nonlinear term, or structural change).
  4. Be aware of hidden assumptions:
    • For instance, regressions with many predictors and flat priors often exaggerate coefficient variability.
    • This mirrors the over-dispersion seen in independent 8-school estimates.
  5. If sensitivity analysis shows large differences, incorporate the alternative models directly into a joint prior
    or explicitly report the dependence of results on untestable assumptions.
  6. In some cases, acknowledge that data are insufficient to answer the question definitively.
    In others, add more information (priors or data) to stabilize the inference.

Essential Idea

Continuous model expansion means viewing models as flexible frameworks, not fixed choices.
Instead of selecting one “true” model, we continuously refine and enlarge models to better reflect both data and knowledge.
Sensitivity analysis complements this by revealing how conclusions about θ change across plausible modeling choices — ensuring that our inferences are robust, transparent, and scientifically grounded.