Graphical posterior predictive checks

1. Goal of graphical checks

Graphical posterior predictive checking is about visually comparing:

1. the observed data $y$, and
2. replicated data
   • $y^{\text{rep}} \sim p(y^{\text{rep}} \mid y) = \int p(y^{\text{rep}} \mid \theta) p(\theta \mid y)\, d\theta$

to see whether the model can generate data that “look like” what we actually observed. If the model is good, the real plot should blend in with the simulated plots. If the real plot looks special, structured, or too clean/too messy compared to the simulations, that signals model misfit.

The text groups graphical checks into three main types:

1. Direct displays of the full data
2. Displays of summaries or batches of parameters
3. Residual-based displays (including binned residuals)

---

2. Direct display of all the data

Example: 3-way binary data (persons × reactions × situations)

• Data: a 3D array of binary responses (yes/no) for 6 people, 15 reactions (rows), 23 situations (columns).
• They plot the actual data in one column.
• Then they generate several replicated datasets $y^{\text{rep}}$ from the fitted logistic regression model and plot those beside it.

Key trick: before plotting, they reorder rows, columns, and persons by increasing average response. That reordering becomes part of the “test statistic” $T(y)$: the display is not just raw numbers, it is a structured view designed to show patterns.

What they found:

• The real data show strong row/column patterns (blocks, almost-all-0 rows, etc.).
• The replicated data from the model look fuzzy and random.
• So the model (a logistic regression for individual responses) fails to capture these systematic person-by-item patterns.

Then they show the same data without ordering. Without the reordering, the misfit is much harder to see. This makes the point: good visualization design is part of model checking — how you arrange the data can make misfit obvious or invisible.

Takeaway: direct side-by-side plots of $y$ and several $y^{\text{rep}}$ are powerful, but only if you display them in a way that highlights structure (sorting, grouping, alignment).

---

3. Displaying summaries or parameter inferences

Sometimes the dataset is large or hierarchical, and it’s more useful to compare batches of parameters to what the model says they should look like.

Example: psychometric hierarchical model

• Model has two sets of parameters:
  • patient-level probabilities $\phi_1, \dots, \phi_{90}$
  • symptom-level probabilities $\psi_1, \dots, \psi_{69}$
    Each was given an independent Beta(2, 2) prior.
• After fitting the model, we have posterior draws of all 159 parameters.
• If the model were correct (data truly came from that prior + likelihood), then a single posterior draw of all $\phi$ or all $\psi$ should look like it was drawn from the assumed prior distribution (because of the joint Bayesian generative story: observed data and posterior draw both come from the same model).

So they plotted histograms of one posterior draw of the 90 patient parameters and one posterior draw of the 69 symptom parameters, and overlaid the Beta(2, 2) curve.

What they saw: way too many values near 0 compared to Beta(2, 2). That means: “our prior assumption that all these parameters are like Beta(2, 2) i.i.d. is not realistic for this dataset.”

What they did: they revised the prior to a mixture of Beta distributions (one spiky near 0, one broader) to better match the empirical shape of the inferred parameters, then refit and re-plotted. The second set of histograms (with the mixture priors) matched much better.

Point: this is also a posterior predictive check — but at the parameter level instead of at the raw-data level. You’re asking: “If I simulate a new batch of patients/symptoms under my model, would I see this same pattern of parameter values?” If not, the model (especially the prior structure) needs refinement.

---

4. Residual plots and binned residual plots

For regression-type models, it’s common to look at residuals to see where the model is off.

Bayesian residuals

• Model has mean function $g(x, \theta) = \mathbb{E}(y \mid x, \theta)$.
• Given one posterior draw $\theta^{(s)}$, define realized residuals:
  • $r_i^{(s)} = y_i – g(x_i, \theta^{(s)}).$
• You can plot these residuals vs. fitted values or vs. predictors, just like classical residual plots.
• Classical residual plots are basically the same thing but using a point estimate $\hat\theta$; Bayesian plots make the dependence on θ explicit.

Problem with discrete data

If $y$ is discrete (like pain scores 0–4), raw residual plots look striped and ugly, because:

$y_i = \text{discrete}, \quad \text{predicted} \approx \text{continuous} \Rightarrow \text{residual} = y_i – \text{predicted}$

can only take a few values. Even if the model is correct, the plot won’t look “flat.”

Solution: binned residual plots

1. Choose a variable to plot against (often predicted mean).
2. Sort observations by that variable.
3. Bin them into K bins with about equal counts.
4. For each bin, compute:
   • average x (or average predicted) → $\bar w_k$
   • average residual → $\bar r_k$
5. Plot $\bar r_k$​ vs. $\bar w_k$​.

Because each point is now an average of several residuals, by the central limit theorem these averages are closer to normal and symmetric around 0, so it’s easier to see systematic patterns.

Then, since we are in a Bayesian model-checking setting, we can also:

• simulate replicated data $y^{\text{rep}}$,
• compute binned residuals for those,
• and draw 95% reference bands.
  If the observed binned residuals go systematically above/below those bands, that’s evidence of misfit (e.g., model underpredicts low scores and overpredicts high scores).

Why binning even though we can simulate? Because binning makes the type of misfit (too low at low x, too high at high x, curvature, etc.) much clearer for a human to read.

---

5. General interpretation

• Any plot you make of the real data can, in principle, be compared to the distribution of the same plot under replicated data from the model.
• The question is always: “Would a plot like this be typical if the model were true?”
• Sometimes people make the wrong comparison — they compare to “pure randomness” instead of “randomness under the actual model.” The chapter reminds us: you must compare to what the fitted model says, not to some vague baseline.

---

6. Overall takeaway

• Graphical posterior predictive checks are just the visual version of posterior predictive checking.
• The steps are the same:
  1. fit model → get $p(\theta \mid y)$
  2. simulate $y^{\text{rep}}$ from $p(y^{\text{rep}} \mid y)$
  3. make the same graph for $y$ and for several $y^{\text{rep}}$
  4. visually check whether the real one looks like the simulated ones.
• Good plotting choices (ordering, grouping, binning) are essential — they often make the difference between “I can’t see any problem” and “the model is clearly missing structure.”