1. What is a “designed experiment” here?

  • An experiment = the experimenter chooses which treatment each unit gets.
  • Because the assignment is under our control, we want to make causal statements: “What would have happened to this unit if it had gotten the other treatment?”
  • We observe only one outcome per unit, but we want to generalize in two directions:
    1. to the outcomes the same units would have had under the other treatment (the missing potential outcomes),
    2. and often to other units in a larger population.

So the whole Bayesian job = combine the model for outcomes with the (known) randomization to get posterior inference.

2. Completely randomized experiment (simplest case)

Assume:

  • There are n units, and n is even.
  • Half get treatment A, half get treatment B.
  • For each unit i, there are two potential outcomes:
    • $y_i^A$​: outcome if i got A
    • $y_i^B$​: outcome if i got B
      So the “complete data” would be an $n \times 2$ matrix $(y_i^A, y_i^B)$.

Stability assumption (SUTVA-ish): what you do to unit i does not change the potential outcomes of the other units. So treatments don’t interfere across units.

3. Causal effect: superpopulation vs finite population

For unit i, the individual causal effect = $y_i^A – y_i^B$​.

We can define the main causal estimand in two ways:

  1. Superpopulation framework
    • Think: these units are draws from a bigger conceptual population.
    • We model $(y_i^A, y_i^B)$ using a likelihood $p(y_i^A, y_i^B \mid \theta)$.
    • Then the average causal effect is $E(y_i^A – y_i^B \mid \theta) = E(y_i^A \mid \theta) – E(y_i^B \mid \theta).$
    • So this estimand is a function of θ. After we get the posterior $p(\theta \mid y_{\text{obs}})$, that induces a posterior for the causal effect.
  2. Finite-population framework
    • Now we care only about these n units.
    • Define $\bar y^A$ = average of the A-potential outcomes of these n, and $\bar y^B$ similarly.
    • Finite-population causal effect = $\bar y^A – \bar y^B$.
    • This connects to survey-sampling style inference.

In practice, people often care about the superpopulation effect, but it’s useful to see how the finite-population view fits in.

4. Inclusion / assignment model (I)

Define an indicator for which treatment each unit actually got:

  • For unit i, $I_i = (I_i^A, I_i^B)$:
    • (1, 0) → got A
    • (0, 1) → got B
    • (0, 0) → not (yet) treated
    • never (1, 1)

In a completely randomized experiment with n/2 in A and n/2 in B:

  • The probability of that assignment is just a known combinatorial constant (all splits with n/2 in A are equally likely).
  • So $p(I \mid y, \phi) = p(I),$ i.e. it does not depend on outcomes. That’s what “ignorable” means here.

Because it’s ignorable, we don’t have to keep conditioning on I in our posterior—we can just analyze the outcome model.

5. Bayesian inference when assignment is ignorable

Since the design is ignorable, we can do:

$p(\theta \mid y_{\text{obs}}) \propto p(\theta) \, p(y_{\text{obs}} \mid \theta)$

That is, we just use the observed outcomes under the treatment they actually got.

Often we model A and B separately:

  • For those with $I_i = (1,0)$: model $p(y_i^A \mid \theta_A)$
  • For those with $I_i = (0,1)$: model $p(y_i^B \mid \theta_B)$

So

$p(y_{\text{obs}} \mid \theta) = \prod_{i: I_i=(1,0)} p(y_i^A \mid \theta_A) \prod_{i: I_i=(0,1)} p(y_i^B \mid \theta_B)$

Then we get posterior draws of $\theta_A, \theta_B$​.

6. Filling in the missing potential outcomes

We only observe one outcome per unit, but the finite-population effect needs both $y_i^A$​ and $y_i^B$​. So we need to predict the missing ones.

Formally we want

$p(y_{\text{mis}} \mid \theta, y_{\text{obs}})$

To do that we need a joint model for $(y_i^A, y_i^B) \mid \theta$, not just separate marginals. That joint model has its own parameters, say $\theta_{AB}$​, which control things like the correlation between the two potential outcomes.

Key point: the likelihood from the actual experiment doesn’t identify $\theta_{AB}$​, because we never observe both $y_i^A$​ and $y_i^B$​ for the same unit. So the posterior for $\theta_{AB}$​ is basically coming from the prior, averaged over the posterior of $\theta_A, \theta_B$​.

So: finite-population inference is more sensitive to assumptions about the joint distribution than superpopulation inference.

7. Large-sample correspondence (why it gets nicer when n is big)

When n is large and the n units are a sample from a large population:

  • The causal estimand for the big population can often be written using only the separate treatment parameters $\theta_A, \theta_B$​.
  • Then the posterior for $\bar y^A – \bar y^B$ is approximately normal:
    • $(\bar y^A – \bar y^B)\mid y_{\text{obs}} \approx N\left(\bar y^A_{\text{obs}} – \bar y^B_{\text{obs}}, \; \frac{2}{n}(s_{A,\text{obs}}^2 + s_{B,\text{obs}}^2)\right)$
    • (that’s the formula they wrote).
  • This looks very much like the usual randomization / sampling-theory result.
  • So: for large samples, Bayesian and classical answers look very similar.

8. More complex designs: randomized blocks, Latin squares

Principle: include in the model every variable that was used to decide treatment assignment. Then the design becomes ignorable.

Latin square example:

  • 25 plots, 5 treatments (A–E).
  • The design uses row and column structure (location).
  • To make the design ignorable, model $p(y \mid x, \theta)$ where x = location info (row, column).
  • Start with an additive linear model on (possibly log) yields: row effect + column effect + treatment effect.
  • You can add interactions if, for example, treatment effect changes across the field.

Important note they make:

  • Even if someone says “actually it was just completely randomized and by chance balanced in rows/columns,” a model that uses location is still better, because it uses real information and will give more precise inference.
  • If you ignore x under complete randomization, inference is still valid, just less efficient.

9. Sequential designs

  • Sometimes treatment probabilities for unit i depend on earlier outcomes or earlier assignments.
  • Then the design is not “strongly ignorable.”
  • But it is ignorable conditional on all the variables used in the treatment-allocation rule: time of entry, earlier outcomes, etc.
  • So the recipe is the same: condition on what the design used.

10. Minimal analysis vs using extra predictors

  • From the design you can figure out the minimal set of covariates you must condition on to make the assignment ignorable. That’s the “adequate summary.”
  • But often you have more information than that (e.g. sex, age, location).
  • In Bayesian analysis it’s usually better to use all relevant info because it improves precision and lets you generalize better.
  • Exception: if some population margins (like % male/female in the population) are not known, adding those covariates requires extra modeling, so sometimes the minimal analysis is more convenient.

11. Cow experiment example (covariate-balanced randomization)

  • 50 cows, 4 diets (treatments), 3 pre-treatment covariates.
  • The researchers kept re-randomizing until the covariates looked balanced.
  • That makes the final assignment ignorable because it only used observed covariates.
  • Minimal analysis: outcome ~ treatment + those 3 pre-treatment covariates.
  • A multivariate model would be even better because there were 6 related outcome measures.
  • If we only care about treatment effects in general (not about exactly those 50 cows), we can just do superpopulation inference.

12. Main takeaways

  1. Ignorability is the key. If treatment assignment depends only on observed variables, we can ignore the assignment mechanism in the likelihood and just model outcomes given those observed variables.
  2. Superpopulation inference is easy in completely randomized experiments:
    • $p(\theta \mid y_{\text{obs}}) \propto p(\theta)\,p(y_{\text{obs}} \mid \theta)$
  3. Finite-population inference needs the joint model for both potential outcomes and is more sensitive because we never see both for the same unit.
  4. Including the factors used in the design (rows, columns, time, covariates) makes more complex designs work the same way as the simple case.
  5. Bigger samples → Bayesian ≈ classical for these causal contrasts.
  6. Everywhere the original had ω, we treat it as θ: it’s just the vector of model parameters.