Bayesian inference requires a model for data collection

1. Why data-collection design matters

Suppose 10 die rolls are shown and all are 6. What to believe about the die depends on how those 10 sixes arose:

1. only 10 rolls were made → suspicious but maybe possible;
2. 60 rolls were made but only sixes were reported → totally biased sample;
3. the plan was “report the first 10 sixes no matter how long it takes” → now 10 sixes could have taken 500 rolls.

In all three cases the observed values are the same, but the distribution of the observed data is different because the reporting/selection mechanism is different. So the model must include not only the data values but also the process that made them the observed ones.

So the statement “Bayesian inference is conditional on the observed data, so design doesn’t matter” is only true if “observed data” is understood to include which parts of the full data were observable and why.

---

2. General pattern: observed vs. complete data

Many applied problems fit the same template:

• there is some bigger set of potential/complete data $y$,
• only some part of it is actually observed,
• the rest is missing by design (not sampled, not assigned that treatment, censored, nonresponse).

Examples:

• sample surveys → observe n out of N units, want to infer about all N;
• experiments → observe outcome under assigned treatment, potential outcomes under other treatments are missing;
• censored/truncated data → observe only some part of the distribution;
• nonresponse → some values intended to be collected are missing.

All of these can be described by:

• a full data array $y$,
• and an indicator III saying which components of $y$ are seen.

So the model should really be for $(y, I)$, not just $y$.

---

3. Why design is helpful

Good designs (random sampling, randomized experiments) make the missingness mechanism simpler. Later this is called ignorable: roughly, if the way data are missing/selected doesn’t depend on the unseen values in a bad way, then the analysis can focus on the outcome model and treat the design part separately or as known.

The key principles stated:

1. Use all relevant information — the pattern of what was observed can itself be informative.
2. Ignorable designs (often randomized) tend to make inferences less sensitive to modeling choices than nonignorable, ad-hoc designs.
3. Conditioning on more variables (e.g. including more covariates that affected selection) can make the analysis more valid, but also more sensitive to how the model for $y \mid x$ is specified.
4. Thinking about what could have been observed (not just what was) helps structure inference for finite-population quantities (population mean, average treatment effect) and for posterior predictive checks, because posterior predictive checks should mimic the same design that produced the real data.

---

4. Big takeaway

Bayesian analysis is not “design-free.” The design tells which parts of the full data space were even allowed to show up. If that is ignored, the posterior can look mathematically fine but answer the wrong real-world question. The fix is to expand the probability model to include the data-collection mechanism — sampling, treatment assignment, censoring, missingness — and then condition on both “what we saw” and “how we came to see it.”