Sensitivity and the role of randomization

1. Big idea

• When the design is ignorable, Bayesian inference is very simple: we just model the observed outcomes and do $p(\theta \mid y_{\text{obs}}) \propto p(\theta)\,p(y_{\text{obs}} \mid \theta).$
• In that case, the posterior for θ and for the missing outcomes $y_{\text{mis}}$ does not depend on the fine details of how we assigned treatments, as long as the design is ignorable.
• So: ignorable design ⇒ posterior is insensitive to design details (given the model).

---

2. Where randomization comes in (no covariates case)

• Suppose we don’t have fully observed covariates $x$.
• Then the only way to make the assignment mechanism $p(I_1, \dots, I_n \mid \phi)$ “symmetric” (invariant to permuting unit labels) and hence ignorable is to randomize.
• So with no covariates, randomization is basically the tool that guarantees ignorability (except degenerate cases like “everyone gets A”).

But even among ignorable designs, some are better:

• Example: 10 subjects, 2 treatments, noninformative prior.
  • Design 1: force 5 in A and 5 in B (complete randomization with balance).
  • Design 2: flip a fair coin for each person independently.
  • Both are ignorable, but Design 1 gives more precise posterior for the average treatment effect because it guarantees balance.
• So: ignorable ≠ equally good. Some ignorable designs give tighter posteriors.

---

3. Randomization when covariates are available

Now suppose we do have fully observed covariates $x$ (e.g. plot locations, phone numbers, age). Then:

• We can make an ignorable design without randomization — a deterministic design — by making the treatment a known function of $x$. That gives propensity scores of 0 or 1.
• So randomized designs are now competing with deterministic-but-ignorable designs.

So what’s the point of randomization here?

They give a field example:

• 12 plots in a row.
• Two fertilizers A and B.
• Two possible designs:
  1. Random: choose 6 plots at random for A.
  2. Systematic: ABABABBABABA (a fixed pattern).
• Both are ignorable given x (the locations).
• A Bayesian would anyway fit $p(y \mid x, \theta)$ — at least with a linear trend in location — for either design.

So what’s the extra value of randomization?

---

4. Hidden advantages of randomization (even in Bayesian analysis)

(a) Model-checking leverage

• Suppose you ran the randomized design but you initially analyzed the data ignoring x.
  • That analysis is still valid, because the randomized design is ignorable even without x.
• Then suppose x becomes available, so you re-analyze with x.
  • You now get $p(y_{\text{mis}} \mid y_{\text{obs}}, x)$, which should be more precise and more appropriate.
  • If the two analyses (without x vs with x) don’t line up in a reasonable way, that tells you your outcome model might be misspecified.
• With a purely systematic design, you don’t automatically get that “second look” unless you start modeling a distribution over x. So randomization gives you an extra diagnostic handle.

(b) Posterior predictive checks in future replications

• If the design was randomized, then hypothetical future experiments will have different treatment allocations.
• That makes it easier to do posterior predictive checks that mimic future experiments.
• With a fixed systematic design, every replication has the same pattern, so you lose some flexibility.

(c) Avoiding systematic dependence on one model

• A single, rigid, systematic design can make your inferences sensitive to a particular modeling assumption about $p(y \mid x, \theta)$.
• If you keep using the same systematic pattern across many experiments, you risk building in a consistent bias relative to that assumption.
• Randomizing patterns across experiments makes the overall set of experiments exchangeable and reduces that systematic dependence.

---

5. Designs that “cheat”

Another reason to like randomization: it makes it harder for the experimenter to bias the results (on purpose or by accident).

Three ways this can show up in Bayesian terms:

1. Assignment depends on unrecorded covariates
   • Example: you secretly give treatment A to sunny plots and B to shaded plots, but you don’t record “sunny vs shaded.”
   • Then the assignment is not ignorable, because it depends on something you didn’t include in the model.
   • You now must model the selection mechanism → hard, and very sensitive.
2. Assignment depends on recorded covariates, but they are confounded with treatment
   • If all sunny plots get A and all shaded get B, then “treatment” and “sunny” are the same variable in the observed data.
   • Then the likelihood can’t tell whether the effect is from treatment or from sun.
   • So even though it’s ignorable in principle (because the covariate is recorded), the inference becomes highly sensitive to how you model $p(y \mid x, \theta)$.
3. Non-distinct parameters (φ and θ dependent)
   • Sometimes the choice of design itself depends on beliefs about the treatment effect.
   • Example: the experimenter uses complete randomization when they expect big effects, but randomized blocks when they expect small effects.
   • Then the design parameter φ and the outcome parameter θ are not independent a priori → ignorability fails.
   • Often data will dominate and you can ignore it in practice, but theoretically it’s no longer cleanly ignorable.

Randomization helps prevent (1) and (2) from creeping in unnoticed.

---

6. Nonrandomized studies

They end by contrasting with a clearly nonrandom design: standing on a street corner 11–12 and interviewing whoever passes.

Two ways to model it:

1. Nonignorable: inclusion depends on unobserved “travel pattern” → must model $p(I \mid y)$.
2. Ignorable (forced): define a fully observed indicator $x_i = 1$ if the person passed by that hour, 0 otherwise. Then we model $p(y \mid x, \theta)$, but we have no data for $x=0$. So inference will be very sensitive to priors unless the sample fraction is large.

So: even if you can mathematically make it ignorable, without randomization the inference can be fragile.

---

7. Final message

• Randomization is not just a frequentist relic; in Bayesian analysis it:
  • guarantees ignorability when no covariates are available,
  • gives better average precision when it enforces balance,
  • gives extra opportunities for model checking,
  • makes posterior predictive checks more flexible,
  • and reduces the chance that unrecorded, experimenter-driven choices secretly drive the inference.
• Our goal is always to get to the “standard Bayesian form”:
  • $p(\theta \mid x, y_{\text{obs}}) \propto p(\theta)\,p(x, y_{\text{obs}} \mid \theta),$ and good, ignorable, preferably randomized designs make that safe and less sensitive.