Example: parallel experiments in eight schools

1. Background and Purpose

Eight different high schools each conducted an independent randomized experiment to measure the effect of short-term SAT-V (Verbal) coaching programs.
Each experiment produced:

• an estimated effect $y_j$ (in SAT-V score points)
• a known standard error $\sigma_j$

for $j = 1, \dots, 8$.

School	Estimated effect $y_j$​	Standard error $\sigma_j$
A	28	15
B	8	10
C	−3	16
D	7	11
E	−1	9
F	1	11
G	18	10
H	12	18

All $y_j$​ were computed using regression models that adjusted for PSAT-V and PSAT-M scores, ensuring comparability across treatment and control groups.
Each school had 30+ students, so it is reasonable to treat $y_j \mid \theta_j \sim N(\theta_j, \sigma_j^2)$

as a good approximation, with known sampling variances.

---

2. Nonhierarchical Analyses

(a) Separate (No Pooling)

If each experiment is analyzed independently:

$\theta_j \mid y_j \sim N(y_j, \sigma_j^2)$

All 95% intervals overlap substantially, so none of the differences are statistically meaningful.
For example, School A’s separate posterior mean is $28$ with $\text{sd} = 15$, implying a 50% chance that the true effect exceeds 28—an implausible claim given the other schools’ results.

(b) Complete Pooling

Assume all eight schools share one common effect $\theta$.
The precision-weighted pooled estimate is:

$\bar{y}_{..} = \frac{\sum_{j=1}^{8} y_j / \sigma_j^2} {\sum_{j=1}^{8} 1 / \sigma_j^2} = 7.7$

with posterior variance:

$V = \left( \sum_{j=1}^{8} \frac{1}{\sigma_j^2} \right)^{-1} = 16.6$

Thus:

$\theta \mid y \sim N(7.7, 16.6), \quad \text{95\% CI: } [ -0.5, 15.9 ]$

The classical ANOVA test for equal effects yields $\chi^2 = \sum_{j=1}^{8} \frac{(y_j – \bar{y}_{..})^2}{\sigma_j^2} = 4.6 < 7 = \text{df}$,

indicating no significant between-school variation.
The ANOVA variance estimate $\hat{\tau}^2 = (MS_B – MS_W)/n$ is even negative, suggesting no detectable heterogeneity.

Still, complete pooling ignores clear differences in observed $y_j$​ values and over-shrinks all schools to the same mean.

---

3. Limitations of the Two Extremes

• Separate analysis: too noisy and over-confident in individual $y_j$​.
• Complete pooling: too restrictive—assumes identical true effects.

For instance, School A’s separate estimate implies $P(\theta_A > 28.4) = 0.5,$

while the pooled model implies $P(\theta_A < 7.7) = 0.5.$

Neither seems reasonable. We need a compromise between the two—partial pooling.

---

4. Hierarchical Bayesian Model

The hierarchical normal model is:

$\begin{aligned} y_j \mid \theta_j &\sim N(\theta_j, \sigma_j^2), \quad j = 1, \dots, 8, \\ \theta_j \mid \mu, \tau &\sim N(\mu, \tau^2), \\ \mu &\sim \text{Uniform}(-\infty, \infty), \\ \tau &\sim \text{Uniform}(0, \infty). \end{aligned}$

Here:

• $\mu$ = overall mean coaching effect (population mean)
• $\tau$ = between-school standard deviation (heterogeneity)

---

5. Posterior Computation

Because

$p(\theta, \mu, \tau \mid y) = p(\tau \mid y) \, p(\mu \mid \tau, y) \, p(\theta \mid \mu, \tau, y)$

sampling proceeds in three steps:

1. Sample $\tau$
   • from the univariate posterior:
     • $p(\tau \mid y) \propto p(\tau) V_\mu^{1/2} \prod_{j=1}^{8} (\sigma_j^2 + \tau^2)^{-1/2} \exp \!\left[-\frac{(y_j – \hat{\mu})^2}{2(\sigma_j^2 + \tau^2)} \right]$
     • where $\hat{\mu} = \frac{\sum_j y_j / (\sigma_j^2 + \tau^2)}{\sum_j 1 / (\sigma_j^2 + \tau^2)}, \quad V_\mu = \left(\sum_j \frac{1}{\sigma_j^2 + \tau^2}\right)^{-1}.$
2. Sample $\mu \mid \tau, y$
   • from $\mu \mid \tau, y \sim N(\hat{\mu}, V_\mu).$
3. Sample each $\theta_j \mid \mu, \tau, y$
   • from $\theta_j \mid \mu, \tau, y \sim N(\hat{\theta}_j, V_j), \quad \hat{\theta}_j = \frac{y_j / \sigma_j^2 + \mu / \tau^2}{1/\sigma_j^2 + 1/\tau^2}, \quad V_j = \frac{1}{1/\sigma_j^2 + 1/\tau^2}.$

They performed 200 simulation draws following this procedure.

---

6. Posterior for $\tau$

The marginal posterior $p(\tau \mid y)$ hows:

• Most probability mass near $\tau = 0$
• Values $\tau > 10$ are less than half as likely as $\tau = 0$
• $P(\tau > 25) \approx 0$

Interpretation: The schools’ true effects are likely similar but not identical.

---

7. Posterior Behavior of $\theta_j$

The conditional posterior means $E(\theta_j \mid \tau, y)$:

• For small $\tau$: all $\theta_j$ cluster around 7–10 points (strong pooling).
• As $\tau$ grows: $\theta_j$​ revert toward the raw $y_j$​ (less pooling).

The conditional posterior standard deviations $\text{sd}(\theta_j \mid \tau, y)$:

• Increase with $\tau$;
• Approach $\sigma_j$​ as $\tau \to \infty$.

Thus, partial pooling reduces uncertainty for small $\tau$, but respects heterogeneity if $\tau$ is large.

---

8. Interpreting School A’s Large Effect

Even though School A’s observed effect is $y_A = 28$:

For plausible $\tau$ values, the posterior probability of such a large true effect is small.
For example, if $\tau = 10$:

$P(\theta_A < 28 \mid \tau=10, y) = \Phi\!\left(\frac{28 – 14.5}{9.1}\right) \approx 0.93$

so there is only a 7% chance the true effect exceeds 28.
Across all schools, these probabilities are ≥ 97%.
Hence, it is unlikely that any true school effect is as large as 28 points.

---

9. Why Not Condition on the Mode of $\tau$

Because the posterior mode of $\tau$ is at 0, fixing $\tau=0$ would yield:

$\theta_j = 7.7 \pm 4.1, \quad \text{for all } j$

which implies complete pooling.
But integrating over $p(\tau \mid y)$ shows that modest heterogeneity exists; the full Bayesian analysis avoids over-shrinkage.

---

10. Posterior Results (from 200 draws)

School	2.5%	25%	Median	75%	97.5%
A	−2	7	10	16	31
B	−5	3	8	12	23
C	−11	2	7	11	19
D	−7	4	8	11	21
E	−9	1	5	10	18
F	−7	2	6	10	28
G	−1	7	10	15	26
H	−6	3	8	13	33

All medians lie between 5 and 10; intervals overlap substantially but are wider than the single pooled interval, allowing for heterogeneity.

---

11. Posterior of Derived Quantities

Because we have posterior draws of all $\theta_j$​, we can compute probabilities for any function of them.

• For the maximum effect:
  • $P\!\left( \max_j \theta_j > 28.4 \mid y \right) \approx \frac{22}{200} = 0.11.$
• For comparing schools A and C:
  • $P(\theta_A > \theta_C \mid y) = \frac{141}{200} = 0.705.$

This illustrates the flexibility of the Bayesian simulation: any comparison or ranking is straightforward.

---

12. Main Takeaways

1. Separate analysis overstates differences.
2. Complete pooling erases real variability.
3. Hierarchical model provides a middle ground—partial pooling—through $\tau$.
4. Posterior results shrink extreme $y_j$​ toward the mean, with stronger shrinkage for noisier estimates.
5. Integrating over $p(\tau \mid y)$ avoids the distortions of fixing $\tau$ at its mode.

In summary, the hierarchical Bayesian model:

$\begin{aligned} y_j \mid \theta_j &\sim N(\theta_j, \sigma_j^2), \\ \theta_j \mid \mu, \tau &\sim N(\mu, \tau^2) \end{aligned}$

yields realistic inference—borrowing strength across schools while properly accounting for uncertainty in both $\mu$ and $\tau$.