Observational studies

1. What makes observational studies different?

• In an experiment, the researcher assigns treatment.
• In an observational (nonexperimental) study, the researcher just observes who got which “treatment” (coaching vs no coaching, program vs no program, drug vs no drug), but didn’t control it.
• The data often look like experimental data (treatment + outcome for each unit), so people analyze them like experiments.
• But: in observational data, the treated and control groups can be very different to begin with, especially in unmeasured ways. That’s the core problem.

Example from the text:

• SAT coaching = experimental if the school randomized students to coaching vs no coaching.
• SAT coaching = observational if students chose coaching themselves.
• In the second case, coached students might already be more motivated → bias.

---

2. When can observational studies still support causal inference?

They list 5 conditions that make an observational study more credible:

1. Good control for background differences: we have enough covariates to explain why some got treated and others didn’t.
2. Enough units in each treatment group: so posteriors aren’t too wide.
3. Design not tailored to the desired answer: the way data were collected isn’t “influenced” by the outcome we want.
4. Missingness handled: dropout, attrition, etc. are either small or modeled.
5. Analysis matches design: you actually condition on the variables that determined treatment.

If these aren’t met, causal conclusions get very sensitive.

---

3. Why sensitivity is worse than in experiments

• In randomized experiments, treated and control groups are balanced in expectation.
• In observational studies, treated and control groups can be unbalanced — e.g. only richer people take the training.
• When there is a lot of imbalance, then adjusting with a regression model $y \mid x, \text{treatment}$ becomes model-dependent:
  • with one functional form, you see a positive effect,
  • with another plausible functional form, the effect disappears.

• That’s exactly what their above image is showing:
  • (a) balanced → treatment effect is stable across models.
  • (b), (c) unbalanced → treatment effect flips depending on whether you fit linear or mildly nonlinear.

So: lack of overlap ⇒ high modeling sensitivity.

---

4. Role of propensity scores

• Propensity score = $e(x) = \Pr(I_i = 1 \mid X_i = x)$: probability of getting the treatment, given covariates.
• In a bad observational study, treated units have propensity scores near 1, controls near 0 → no overlap.
• If there is little or no overlap in propensity scores, then causal estimates are fragile — exactly like in their pictures.
• Using propensity scores (matching, trimming, subclassification) can:
  • restrict analysis to the region where treated and control look similar,
  • and therefore reduce sensitivity to the exact outcome model.

So in Bayesian observational analysis, estimating the propensity score is often a diagnostic step: it tells you how bad the imbalance is.

---

5. Bayesian view for observational studies

What do we do in Bayesian analysis?

1. Collect a lot of covariates so that treatment assignment is (approximately) ignorable given those covariates.
   • “Ignorable” here = $\Pr(I \mid y, x, \phi)$ depends only on $x$, not on unobserved potential outcomes.
2. If we accept ignorability, we can analyze the observational data as if it were an experiment with non-equal assignment probabilities:
   • $p(\theta \mid x, y_{\text{obs}}) \propto p(\theta)\,p(y_{\text{obs}} \mid x, \theta)$
3. Then we can do the usual Bayesian things: regression adjustment, poststratification, averaging over covariate distribution.

But: if we cannot make treatment “almost ignorable” with available covariates, then

• inferences become very sensitive,
• and sometimes the data basically can’t answer the causal question.

That’s their warning sentence: “without enough covariates to make the design approximately ignorable, sensitivity … can be so great that the observed data may provide essentially no information.”

---

6. Tools to reduce sensitivity in observational studies

They list standard design/analysis tricks that all have the same philosophy: constrain the x-space so you don’t have to extrapolate much.

• Matched sampling: pair treated and control units with similar x.
• Subclassification / stratification / blocking: group by x (or by propensity score) and compare within groups.
• Poststratification (very natural in Bayesian models): model within cells and average over population cell proportions.
• Analysis of covariance (regression adjustment): model y∣x,treatmenty \mid x, \text{treatment}y∣x,treatment and then average.

All of these try to make it so that the model for $y \mid x$ is used locally, not to extrapolate from rich to poor or from old to young when only one group got treated.

---

7. Two main difficulties they highlight

1. Unbalanced treatments
   • If good students choose coaching and weak students don’t, and you don’t have enough x to fully adjust, then your result depends a lot on assumptions (e.g. linearity, additivity).
   • Sometimes that makes the dataset basically unusable for the causal question.
2. Unknown / unmeasured drivers of treatment
   • In real observational data, people take treatment for reasons we don’t fully observe (motivation, health, mood).
   • Then treatment is not ignorable, and your posterior for θ will depend on how you model that nonignorable mechanism.

So again: data collection quality matters a lot more than in randomized experiments.

---

8. Principal stratification (intermediate variables on the causal path)

Now they shift to a special but very common situation: randomized experiment with noncompliance (or, more generally, an intermediate post-treatment variable).

Set-up:

• $I$: assignment (1 = assigned treatment, 0 = assigned control)
• $C$: intermediate post-assignment variable (e.g. did the person actually take the vitamin?)
• Final outcome: $Y$

Naive mistake: treat observed $C_{\text{obs}}$​ as if it were a baseline covariate and stratify on it. That’s wrong because $sC_{\text{obs}}$​ depends on assignment.

Correct idea: define principal strata by the pair of potential intermediate outcomes $(C(1), C(0))$:

• “Compliers”: $C(1)=1, C(0)=0$
• “Never-takers”: $C(1)=0, C(0)=0$
• (In other cases you can also have always-takers or defiers, depending on the study.)

These principal strata are pre-treatment characteristics conceptually — they don’t change with assignment — so it’s legitimate to stratify on them.

That’s what “principal stratification” means.

---

9. Example: vitamin A experiment with noncompliance

• Vitamin A was available only to those assigned to treatment.
• So: control group can’t get it → everyone has $C(0)=0$.
• Among those assigned treatment, some actually took it (compliers), some didn’t (never-takers).
• So there are two relevant strata: compliers and never-takers.
• The table they show is just counts by (assignment, exposure, outcome).

Goal: estimate the causal effect of actually taking vitamin A for compliers. This is called:

• CACE = Complier Average Causal Effect.
• Also called LATE (Local Average Treatment Effect) in econometrics.

---

10. Link to instrumental variables

In this noncompliance setting:

• Assignment III is randomized → it’s a valid instrument.
• Vitamin actually taken is the treatment actually received.
• Outcome is infant survival.

They give the key identity:

$\text{ITT} = Y^1 – Y^0 = p_c \cdot \text{CACE} + (1 – p_c)\cdot \text{NACE}$

where

• ITT (intention-to-treat effect) = effect of being assigned treatment,
• $p_c$​ = proportion of compliers,
• CACE = effect among compliers,
• NACE = effect among never-takers.

If we make the exclusion restriction:

• assignment has no effect on never-takers (NACE = 0),
  then

$\text{CACE} = \frac{\text{ITT}}{\hat p_c}$

That’s the instrumental variables (IV) estimate: divide the ITT by the compliance rate.

---

11. Bayesian causal inference with noncompliance

They say: the IV formula is nice and simple, but it’s a method-of-moments style estimate.

The Bayesian way:

• treat the compliance status of the control group as missing data,
• write a full model for outcomes, assignment, and compliance,
• sample from the posterior using MCMC,
• and you can relax exclusion restriction if you want.

This is a good example of their overall message: when design is not perfectly clean (observational, noncompliance, intermediate variables), Bayesian modeling lets you write down the full data-generating story — but your results will be as sensitive as your assumptions.

---

12. Core message of this section

• Observational studies can be analyzed “like experiments” only if you make treatment assignment (approximately) ignorable by conditioning on rich covariates.
• Even then, poor covariate overlap ⇒ high sensitivity to functional-form assumptions.
• Propensity scores and design-like devices (matching, stratifying, blocking) are ways to reduce that sensitivity by restricting to comparable units.
• Principal stratification is the right way to handle intermediate post-treatment variables (like compliance).
• In noncompliance settings, the CACE / IV logic gives you the causal effect for people who actually take treatment when offered.