1. Motivation — Why we must model data collection
Bayesian inference doesn’t just model values of the data; it also must account for how those values were obtained.
When some data are missing (e.g., unobserved survey responses, censored data, or untested treatments), we must separate:
- Complete data — what we would have observed in an ideal world, if every possible value were known.
- Observed data — the actual subset we have in hand.
- Missing data mechanism — the process that determines which parts of the complete data were observed.
So, instead of just one random variable $y$, the model now includes:
- $y = (y_1, \dots, y_N)$: the complete data (potential values).
- $I = (I_1, \dots, I_N)$: the inclusion or observation indicators, where $I_i = 1$ if $y_i$ is observed and $I_i = 0$ if it’s missing.
The observed data are $(y_{\text{obs}}, I)$.
The missing data are $y_{\text{mis}} = \{y_i : I_i = 0\}$.
The sample space for inference is thus the product space of $y$ and $I$:
we are modeling both the data values and the observation process jointly.
2. Stability assumption
We assume stability, meaning:
The process that determines whether a value is observed does not change the value itself.
Formally:
$y \text{ is fixed (or generated from } p(y|\theta)) \text{ independently of the observation vector } I.$
If III can actually affect $y$ — for example, fertilizer leaking between plots in an agricultural field experiment — then we must expand the definition of the “complete data” to include all possible outcomes under all possible inclusion patterns (that’s essentially causal inference territory, handled with potential outcomes).
So the stability assumption allows us to write $y$ and $I$ as conceptually separate random variables:
one gives the data, the other tells which parts we see.
3. Including fully observed covariates $x$
We often have other variables that are fully known for all units, like age, gender, or block assignment.
Call these $x$.
They can serve several purposes:
- Substantive modeling — we might care about $p(y|x, \theta)$ itself (e.g., regression coefficients).
- Prediction — even if we only want to model $y$, knowing $x$ improves predictions of missing $ y$.
- Data collection mechanism — the way data are missing might depend on $x$. For example:
- in stratified sampling, selection probabilities depend on strata (a function of $x$),
- in a blocked experiment, assignment depends on blocks.
So, $x$ must be part of the full model whenever III depends on it.
4. Building the joint model
The joint model for data and observation process is factorized as:
$p(y, I \mid x, \theta, \phi) = p(y \mid x, \theta)\; p(I \mid x, y, \phi)$
Where:
- $p(y \mid x, \theta)$ — the data (or outcome) model, describing how the full data are generated.
- $p(I \mid x, y, \phi)$ — the inclusion model, describing how missingness occurs.
- $\theta$ — parameters of the data model.
- $\phi$ — parameters governing the missingness mechanism.
This decomposition expresses two conditional processes:
- Generate the full data $y$ from $p(y|x, \theta)$.
- Then generate the observation pattern III from $p(I|x, y, \phi)$.
This mirrors the intuitive story:
→ “First, nature creates data; then, the design determines which parts you see.”
5. Observed-data likelihood
In real life, we only observe $(y_{\text{obs}}, I)$, not $y_{\text{mis}}$.
So the observed-data likelihood for inference is obtained by integrating over missing values:
$p(y_{\text{obs}}, I \mid x, \theta, \phi) = \int p(y, I \mid x, \theta, \phi) \, dy_{\text{mis}} = \int p(y \mid x, \theta) p(I \mid x, y, \phi) \, dy_{\text{mis}}.$
This is the correct likelihood function that reflects both what values we saw and how we saw them.
6. Posterior distribution
The joint posterior for the model parameters $(\theta, \phi)$ given observed data is:
$p(\theta, \phi \mid x, y_{\text{obs}}, I) \propto p(\theta, \phi \mid x)\; p(y_{\text{obs}}, I \mid x, \theta, \phi)$
Expanding the right-hand side:
$p(\theta, \phi \mid x, y_{\text{obs}}, I) \propto p(\theta, \phi \mid x) \int p(y \mid x, \theta) p(I \mid x, y, \phi)\, dy_{\text{mis}}$
7. Posterior for θ alone
If we are mainly interested in θ (the data model parameters), we integrate φ out:
$p(\theta \mid x, y_{\text{obs}}, I) = \int p(\theta, \phi \mid x, y_{\text{obs}}, I)\, d\phi$
Expanding this yields the key form:
$p(\theta \mid x, y_{\text{obs}}, I) \propto p(\theta \mid x) \int p(\phi \mid x, \theta) \int p(y \mid x, \theta)\, p(I \mid x, y, \phi)\, dy_{\text{mis}}\, d\phi$
This is the general Bayesian inference when missingness may depend on unobserved data.
8. Finite-population vs. superpopulation inference
Two kinds of inferences are possible:
| Type | Definition | Example |
|---|---|---|
| Finite-population | Statements about the specific observed population (e.g., all N individuals) | “What is the total income of all 804 towns in NY?” |
| Superpopulation | Statements about the underlying generative distribution (future or hypothetical data) | “What is the average town population if we drew new towns from this process?” |
The finite-population estimand is often predictable if we can impute the missing pieces $y_{\text{mis}}$.
We can do that by multiple imputation:
- Draw $(\theta, \phi)$ from their posterior.
- Draw $y_{\text{mis}} \sim p(y_{\text{mis}} \mid x, y_{\text{obs}}, I, \theta, \phi)$.
- Combine $y_{\text{obs}} + y_{\text{mis}}$ to estimate totals, means, etc.
In contrast, superpopulation inference concerns properties of θ itself (e.g., regression coefficients, population-level parameters).
It depends more heavily on model assumptions and is therefore less robust.
9. Predictive Distributions
There are two main predictive settings in Bayesian data analysis, depending on whether we are predicting future complete data or future observed data that also reflect the data-collection process.
| Type | Depends on | Meaning / Description |
|---|---|---|
| Future complete data $\tilde{y}$ | $p(\tilde{y} \mid x, \theta)$ | Predicts the distribution of new data values as if all potential outcomes could be fully observed. It depends only on the complete-data model $p(y \mid x, \theta)$ and the posterior for $\theta$. |
| Future observed data $\tilde{y}_{\text{obs}}$ | $p(\tilde{y}_{\text{obs}} \mid x, \theta, \phi) = \int p(\tilde{y} \mid x, \theta)\, p(I \mid x, \tilde{y}, \phi)\, d\tilde{y}_{\text{mis}}$ | Predicts what will actually be observed, accounting for the data-collection mechanism (e.g., sampling, censoring, or missingness). It integrates over unobserved parts and depends on both $\theta$ and $\phi$. |
The second type depends on the data-collection mechanism $p(I|x,y,\phi)$.
This is important for posterior predictive checks: simulated replications should mimic the same design that produced the observed data.
10. Ignorability — when the data-collection model can be ignored
Suppose we ignore the data-collection mechanism and compute:
$p(\theta \mid x, y_{\text{obs}}) \propto p(\theta \mid x)\, p(y_{\text{obs}} \mid x, \theta)$
where
$p(y_{\text{obs}} \mid x, \theta) = \int p(y \mid x, \theta)\, dy_{\text{mis}}$
When is this valid?
We need two conditions:
(1) Missing at Random (MAR)
$p(I \mid x, y, \phi) = p(I \mid x, y_{\text{obs}}, \phi)$
That is: given the observed part of the data (and $x$), the probability of being missing does not depend on the unobserved part $y_{\text{mis}}$.
Intuitively: the missingness depends only on known quantities.
Example:
- In a survey, the probability of response depends on a person’s age (x) but not on their true income (unobserved y) after controlling for age.
- In a clinical study, dropout depends on previous measured symptoms but not on future unobserved outcomes.
Note: MAR can even hold for deterministic rules.
E.g., “audit all taxpayers with declared income > \$1 M” is MAR, because inclusion depends only on an observed covariate (declared income).
(2) Distinct parameters
$p(\phi \mid x, \theta) = p(\phi \mid x)$
The parameters governing missingness (φ) are independent of those governing the data model (θ) in the prior.
If both MAR and distinct parameters hold,
then the missingness mechanism is ignorable, meaning:
$p(\theta \mid x, y_{\text{obs}}, I) = p(\theta \mid x, y_{\text{obs}})$
So we can safely omit $p(I|x,y,\phi)$ from the likelihood.
11. Classes of Data-Collection Designs
| Design Type | Definition | Examples | Ignorable? |
|---|---|---|---|
| Ignorable & known, no covariates | $p(I \mid x, y, \phi) = p(I)$. The selection mechanism is fixed and unrelated to either $y$ or $x$. | Simple random sampling, completely randomized experiment | Ignorable |
| Ignorable & known given covariates | $p(I \mid x, y, \phi) = p(I \mid x)$. The inclusion or assignment depends only on fully observed covariates $x$; analysis must condition on $x$. | Stratified sampling, randomized block designs | Ignorable (given $x$) |
| Strongly ignorable | $p(I \mid x, y, \phi) = p(I \mid x)$, and treatment assignment is independent of potential outcomes given $x$. | Randomized treatment assignments within covariate groups | Ignorable (given $x$) |
| Ignorable but unknown | The mechanism is not fixed but depends only on fully observed covariates $x$; $\phi$ is unknown but distinct from $\theta$. | Observational studies where doctors assign treatments based on patients’ age or health | Ignorable (given xxx) |
| Nonignorable & known | The mechanism is known (or assumed known) but depends on $y$, including unobserved parts. | Censoring at known thresholds, truncation, rounding, heaping | Not ignorable — must explicitly model $p(I \mid y)$ |
| Nonignorable & unknown | The mechanism depends on unobserved $y$ and is not specified; parameters $\phi$ and $\theta$ are not distinct. | Self-selection bias, survey nonresponse correlated with true unobserved values | Not ignorable — highly sensitive; must model jointly |
12. Propensity scores
In strongly ignorable designs, each unit has a known probability of inclusion or treatment assignment:
$\pi_i = \Pr(I_i = 1 \mid x_i)$
The vector $\pi = (\pi_1, …, \pi_N)$ is the propensity score.
- Conditioning on $\pi_i$ instead of full $x_i$ can simplify the model.
- It makes inference more robust and less dependent on exact modeling of multivariate covariates.
- But: propensity scores are insufficient for posterior predictive checks, since two different designs may have identical propensity scores but different structure (e.g., random vs. block randomization).
13. Unintentional missing data
This includes:
- survey nonresponse,
- dropout in experiments,
- missing values in observational studies.
When the proportion of missing data is small and plausibly “missing at random” given observed covariates, you can treat the mechanism as ignorable.
But when missingness is large (say > 30–50%), the MAR assumption becomes fragile, and modeling $p(I|y,x,\phi)$ directly becomes necessary.
Example:
In causal inference, each subject can only receive one treatment → half of the potential outcomes are structurally missing.
Thus, you must either assume ignorability (treatment independent of potential outcomes given covariates) or model the selection process explicitly.
14. Key Conceptual Hierarchy
| Concept | Description |
|---|---|
| Complete-data model $p(y \mid x, \theta)$ | The probability model describing how all possible data (both observed and unobserved) are generated — the fundamental data-generation process. |
| Inclusion model $p(I \mid x, y, \phi)$ | The model describing the data-collection or missingness mechanism — how the observation indicators III arise given the data $y$. |
| Observed-data likelihood $p(y_{\text{obs}}, I \mid x, \theta, \phi) = \int p(y \mid x, \theta)\, p(I \mid x, y, \phi)\, dy_{\text{mis}}$ | The actual likelihood used for inference, integrating over missing values to combine the data model and inclusion model. |
| Ignorable design | Occurs when $p(I \mid x, y, \phi)$ does not depend on unobserved data $y_{\text{mis}}$ and when parameters $\theta$ and $\phi$ are distinct (independent). In this case, the data-collection mechanism can safely be ignored. |
| MAR (Missing At Random) | Missingness depends only on observed values $y_{\text{obs}}$ and covariates $x$, but not on the missing values $y_{\text{mis}}$. |
| MNAR (Missing Not At Random) | Missingness depends directly on unobserved (missing) values, so the missing-data mechanism must be explicitly modeled. |
| Strong ignorability | The missingness or treatment assignment depends only on fully observed covariates $x$ and is independent of potential outcomes. This allows valid causal inference conditional on $x$. |
15. Summary of intuition
- Bayesian inference requires modeling both the data and how they were obtained.
Ignoring the design is only safe if the mechanism is ignorable (MAR + distinct parameters). - Ignorable designs (like random sampling or randomization) are a gift — they let you analyze just $p(y|x,\theta)$.
- Nonignorable designs make inferences sensitive to modeling assumptions and require explicitly modeling $p(I|x,y,\phi)$.
- Conditioning on covariates used in the design often restores ignorability.
Always include the variables that determined who was observed. - Propensity scores summarize those covariates when the mechanism is strongly ignorable.
- Posterior predictive checks depend on the design — you must simulate replications using the same observation mechanism.
- When missingness is large or nonrandom, your conclusions depend more on priors and the inclusion model than on data alone.
In one sentence:
In Bayesian data analysis, the validity of inference hinges not only on modeling the data distribution $p(y|x,θ)$ but also on properly accounting for the mechanism $p(I|x,y,φ)$ that made those data observable — and the mechanism can be safely ignored only when it is missing at random and parameter-distinct.