In any applied Bayesian problem, there is always additional knowledge that is not formally represented in either the prior distribution or the likelihood function—usually for reasons of convenience, simplicity, or objectivity.
If this extra information suggests that the resulting posterior inferences are clearly inconsistent with known facts, then this points to an opportunity to improve the probability model for both the parameters and the data-generation process.

We illustrate this point through an example involving a hierarchical regression model.

Example: Evaluating Election Predictions through Substantive Political Knowledge

Figure 6.1 shows a forecast made in early October 1992 for the U.S. presidential election in November of that year.
The model produced posterior probabilities that Bill Clinton would win each state, based on a hierarchical linear regression model applied to state-level polling data.

For each state, the height of the shaded bar in the box represents the estimated probability of a Clinton victory, while the width of the box is proportional to the number of electoral votes for that state.

Even before the election took place, some of these forecasts appeared implausible when compared to contemporaneous polling data.
For instance, it was well known at the time that Clinton was performing much more poorly in Texas and Florida than the model suggested.

This does not mean that the model was useless, but it illustrates the importance of knowing where and how the model performs poorly.
After the election, one can perform an even more detailed evaluation by comparing predictions to actual outcomes, identifying specific weaknesses in the hierarchical model.

We will revisit this election forecasting problem later, where it is analyzed in greater detail as an example of a hierarchical linear model.

External Validation

A more formal way to evaluate a model is through external validation—that is, using the model to predict new or future data, and then comparing those predictions with the subsequently observed outcomes.

If the model is well-calibrated, then:

  • Posterior means should, on average, match the true values.
  • Approximately 50% posterior intervals should contain the true values about half the time, and so forth.

External validation was used, for example, to test the empirical probability estimates in the record-linkage study discussed, and again to evaluate a toxicology model later.
In the latter case, the external validation revealed generally good agreement between predictions and observed data, but also some clear discrepancies.

In many applications, however, it is impractical to wait for new data or future outcomes before checking the model.
Therefore, in the remainder of this chapter and the next, we discuss approximate validation methods that use the existing data to mimic the logic of external validation.
These methods form the foundation for posterior predictive checking.

Choices in Defining Predictive Quantities

A single model can be used to make several different types of predictions, depending on what aspects of the process we wish to evaluate.

For instance, in the eight-schools example, possible predictive targets include:

  • The joint prediction for new data from the same eight schools, $p(\tilde{y} \mid y)$;
  • The joint prediction for a new set of eight schools, $p(\tilde{y}_i \mid y), \, i = 9, \ldots, 16$; or
  • Any other combination of predictions for existing and new groups.

Other research contexts can involve even broader predictive targets.
For example, in the analysis of sample surveys or designed experiments, it may be appropriate to consider hypothetical replications of the experiment with new randomizations of selection or treatment assignment—analogous to classical randomization tests.

We will introduce posterior predictive checking, which uses global summaries to evaluate the joint posterior predictive distribution $p(\tilde{y} \mid y)$.
We also discuss extensions that aggregate local predictive checks for individual observations, $p(\tilde{y}_i \mid y)$, an approach closely related to cross-validation methods explored further.

Summary:
Evaluating whether the inferences from a Bayesian model make sense requires combining formal predictive validation with substantive domain knowledge.
By comparing model-based predictions to both external data and realistic expectations, analysts can identify weaknesses, refine assumptions, and build models that more accurately reflect the underlying scientific or social processes.