Exchangeability and hierarchical models

Basic setup

We have $J$ experiments, $j = 1, \dots, J$.

• data for experiment $j$: $y_j$
• experiment-specific parameter: $\theta_j$
• likelihood: $p(y_j \mid \theta_j)$

Example: in a normal model where all groups share a variance but have their own means, you could have $\theta_j = (\mu_j, \sigma^2)$.

We want one joint model for all $\theta_1, \dots, \theta_J$​.

---

1. Exchangeability

Idea: if, before seeing the data, you have no reason to treat group 3 differently from group 12, then your prior for $(\theta_1, \dots, \theta_J)$

should be symmetric in these indices. That symmetry is called exchangeability: $p(\theta_1, \dots, \theta_J) \text{ is unchanged if you permute the indices.}$

In the rat-tumor story: we don’t know anything about which lab or year (other than sample sizes), so it’s natural to treat the 71 studies as exchangeable.

They make a nice point: ignorance ⇒ exchangeability. The less you know to distinguish units, the more exchangeable they should be.

---

2. How to model exchangeability

The standard way (and the one we almost always use) is:

1. Introduce hyperparameters $\phi$ (these describe the population of groups),
2. Given $\phi$, make the groups independent and identically distributed:

$p(\theta \mid \phi) = \prod_{j=1}^J p(\theta_j \mid \phi). \tag{1}$

3. Since $\phi$ is unknown, average over it:

$p(\theta) = \int \left( \prod_{j=1}^J p(\theta_j \mid \phi) \right) p(\phi) \, d\phi.$. (2)

This “mixture of i.i.d. distributions” is exactly what de Finetti’s theorem says you get (in the infinite case): every nice exchangeable sequence can be written like (2).

So: exchangeable ⇒ hierarchical.

• level 1: data $y_j \mid \theta_j$
• level 2: parameters $\theta_j \mid \phi$
• level 3: hyperparameters $\phi$

---

3. The divorce-rate thought experiment (why exchangeability matters)

They use an example with 8 U.S. states’ divorce rates in 1981:

• If you don’t know which state is which, the 8 numbers $y_1, \dots, y_8$​ should be exchangeable.
• After you see 7 of them, you form a posterior predictive for the 8th; because of exchangeability, you expect it to be similar to the ones you saw (say 5–8 per 1000).
• Then they add partial information (“the 8 states are the Mountain states, but you don’t know which is which”): now your prior for the 8 numbers is wider (because you expect an unusually low Utah and an unusually high Nevada), but it is still exchangeable because you still don’t know which is which.
• Finally, if they tell you “the missing state is Nevada,” exchangeability breaks — now state 8 is special, so you don’t treat it symmetrically anymore.

That’s the punchline: exchangeability is about what you do and don’t know. As soon as you get info that distinguishes units, you should encode it, and pure exchangeability stops being appropriate.

---

4. Partial / conditional exchangeability

Often we do know something about the groups — lab, year, previous rate, region, sample size, etc. Then we don’t say $(\theta_1, \dots, \theta_J) \text{ are exchangeable,}$

we say $(\theta_1, \dots, \theta_J) \text{ are exchangeable **given** their covariates } x_1, \dots, x_J.$

That is, $p(\theta_1, \dots, \theta_J \mid x_1, \dots, x_J) = \int \prod_{j=1}^J p(\theta_j \mid x_j, \phi)\, p(\phi \mid x)\, d\phi.$

This is just “do regression in the prior”: groups with similar $x_j$​ will tend to have similar $\theta_j$​, but all are still tied together through $\phi$.

In the rat example, the only extra info is the sample size $n_j$, and that probably doesn’t explain tumor risk, so they stuck with full exchangeability.

---

5. Objections to exchangeability

A natural objection is: “but the 71 rat studies are not identical — different labs, times, cages, etc.”

Their answer: that’s fine. Exchangeability doesn’t mean the $\theta_j$​’s are equal; it just means “we’re going to model them as if they’re draws from the same population because we have no better way to order or group them.” If you do have better information (lab, year), put it into $x_j$​ and use conditional exchangeability.

So the real mistake is not exchangeability; the real mistake is failing to include known relevant covariates.

---

6. Full Bayesian hierarchical model

Before, in the rats, we pretended $(\alpha, \beta)$ of the Beta prior were known. That was convenient but unrealistic.

The full model is:

1. Hyperprior on $\phi$ (here $\phi = (\alpha, \beta)$):
   • $\phi \sim p(\phi)$
2. Group-level (population) model:
   • $\theta_j \mid \phi \sim p(\theta_j \mid \phi), \quad j=1,\dots,J.$
3. Data level: $y_j \mid \theta_j \sim p(y_j \mid \theta_j).$

Joint prior:

$p(\phi, \theta) = p(\phi)\, \prod_{j=1}^J p(\theta_j \mid \phi)$

Posterior:

$p(\phi, \theta \mid y) \;\propto\; p(\phi)\, \prod_{j=1}^J p(\theta_j \mid \phi)\, p(y_j \mid \theta_j).$ (3)

That’s the standard hierarchical Bayes factorization.

Key detail: the data $y$ depend on $\phi$ only through the lower-level parameters $\theta$, so we can write $p(\phi, \theta \mid y) \propto p(\phi)\, p(\theta \mid \phi)\, p(y \mid \theta),$

not $p(y \mid \phi, \theta)$.

---

7. Choosing a hyperprior

We now have to pick $p(\phi)$. Options:

• weak / diffuse prior on $\phi$
• or a proper prior in a plausible region

But: if we make $p(\phi)$ improper, we must check that the posterior is proper, because hierarchical models can blow up at the hyperlevel.

In the rats example: $\phi = (\alpha, \beta)$ controls the Beta population of tumor rates; in the next section they show one way to make a reasonable hyperprior.

---

8. Posterior predictive in hierarchical models

Two natural predictions:

1. More data from an existing group $j$:
   • sample $\theta_j$​ from its posterior, then sample new $\tilde y$​ from $p(\tilde y \mid \theta_j)$.
2. A brand-new group (a new rat experiment):
   • sample $\phi$ from its posterior,
   • sample a new $\tilde \theta \sim p(\theta \mid \phi)$,
   • then sample $\tilde y \sim p(y \mid \tilde \theta)$.

These two are different because the first one conditions on an existing $\theta_j$​, the second one averages over the population.

---

So the core message of this section:

• “We have many related groups” + “we can’t distinguish them much” → assume exchangeability.
• Exchangeability + uncertainty about the population → hierarchical model.
• Then do full Bayes on both the group parameters $\theta_j$ and the hyperparameters $\phi$.