1. Context: why incentives, why Bayesian?

  • Surveys often have low response rates (NYC Social Indicators Survey was < 50%).
  • Incentives (cash/gift, prepaid/postpaid, different amounts) can raise response rate.
  • But incentives cost money.
  • So the real question is decision-like: “For our survey, is \$5 prepaid worth it? What about \$10 postpaid?”
  • To answer that, we need predictions of how much a given incentive will increase response, conditional on survey type (phone vs face-to-face, high vs low burden).
  • We have historical experimental data from many different surveys → that’s why they do a Bayesian hierarchical meta-analysis.

2. Data they have

  • 39 surveys.
  • Each had an embedded experiment with different incentive conditions.
  • Altogether: 101 “conditions” (i.e. specific incentive setups) across those 39 surveys.
  • To simplify, they define 62 differences:
    • $z_i = y_i – y_i^0$​ where $y_i^0$​ is the baseline/lowest incentive in that same survey.
      • This subtracts off the survey-level baseline.
      • So we focus on “increase in response rate due to incentive” rather than absolute response rate.

Above image shows:

  • x-axis: increased dollar amount
  • y-axis: increase in response rate
  • separate panels for mode (telephone vs face-to-face) and burden (high vs low)
  • solid = prepaid, open = postpaid
  • visually: incentives help; prepaid tends to help a bit more.

3. Why a hierarchical model, not a single flat regression?

A plain regression of $z_i$ on amount, timing, form, mode, burden would run into problems:

  1. Sparsity & interactions: some interactions (e.g. prepaid × high burden) are plausible but not well-populated.
  2. Clustering: the 101 conditions belong to 39 different surveys → need random effects.
  3. Unequal sample sizes: some conditions had 80 people, some 2000; we don’t want to just “weight by N” and ignore model error.

So they set up a two-level model.

4. The hierarchical model (their actual setup)

Level 1 (binomial sampling):

$n_i \sim \text{Binomial}(N_i, \pi_i), \quad y_i = \frac{n_i}{N_i}$

Level 2 (regression on incentive features):

$\pi_i \sim N(X_i \beta + \alpha_{j(i)}, \sigma^2)$

  • $X_i$​: encodes
    • Incentive given? (indicator)
    • Value (in tens of 1999 dollars)
    • Timing (pre vs post)
    • Form (cash vs gift)
    • Mode (phone vs face-to-face)
    • Burden (high vs low)
    • and key interactions
  • $\alpha_{j(i)}$​: survey-level random effect for survey $j$, because different surveys have different baselines
  • $\sigma$: leftover (within-survey) noise

Level 3 (random effects):

$\alpha_j \sim N(0, \tau^2)$

Priors:

flat / noninformative for $\beta, \sigma, \tau$.

They stay on the untransformed probability scale (not logit) because in this application response rates aren’t near 0 or 1, so linear is fine and more interpretable.

5. Why include “not significant” interactions?

Because the goal is prediction for a specific future survey design, not hypothesis testing.

Example of needed interactions:

  • Incentive × Value
  • Incentive × Timing
  • Incentive × Burden
  • (and a few 3-way versions)

Even if an interaction isn’t individually significant, if you leave it out you might mis-predict for, say, a low-burden telephone survey with prepaid $10—which is exactly the kind of scenario you need to evaluate.

So the decision-analytic view says:

“Does including this predictor improve our ability to estimate costs/benefits for the decision we care about?”
not
“Is the p-value < 0.05?”

That’s the key philosophical point they’re making.

6. What the fitted model found (their main qualitative results)

From the fitted hierarchical regression:

  • Any incentive: about +3 percentage points on average.
  • Extra $10: +3–4 percentage points.
  • Cash vs gift: cash ≈ +1 point over gift.
  • Prepaid vs postpaid: prepaid ≈ +1–2 points over postpaid.
  • High-burden surveys: incentives work more (≈ +5 points more than for low burden).
  • $\sigma$ (within-survey SD): ~3–4 percentage points → you can’t predict each experimental arm perfectly.
  • $\tau$ (between-survey SD): ~18 percentage points → baseline response rates vary a lot across different surveys.

That last bit is important: big between-survey variation is exactly why we need a hierarchical model and survey-level effects.

7. Turning predictions into a decision for the NYC survey

Target survey:

  • low-burden
  • telephone
  • we prefer cash
  • we must choose: prepaid vs postpaid, and the amount

Complication:

  • Prepaid: must send to everyone we dial, even those who never pick up / screen out → higher outlay.
  • Postpaid: pay only to actual respondents → lower outlay, even if per-dollar effect is slightly smaller.

So they do this calculation:

  1. Use the fitted model to predict increase in response rate for a given incentive option.
  2. From that increased response rate, back-calculate how many calls are needed to get the target number of completed interviews.
  3. Compute cost of incentives:
    • $\text{incentive cost} = (\text{incentive amount} + \$1.25\text{ handling}) \times \text{\# people who get incentive (for postpaid: \#people = \#completed; for prepaid: \#people = everyone contacted/sent)}$
  4. Compute savings in calling from having to complete fewer calls.
  5. Net cost = incentive cost – calling savings.
  6. Divide by #responses to get “extra \$ per completed response” for that incentive option.

They show a concrete example:

  • \$5 postpaid for the caregiver survey (1500 responses target):
    • model says +1.5% response (38.9% → 40.4%)
    • to get 1500 completes at 40.4% → need 3715 eligible households (vs 3856 actually)
    • backing out to #calls → saves about 3377 calls
    • postpaid incentive cost = 1500 × $6.25 = $9375
    • call savings = \$2634
    • net = \$9375 − \$2634 = \$6741
    • per interview extra cost = \$6741 / 1500 ≈ $4.49 per interview to buy that +1.5%

They repeat this for different dollar amounts and for prepaid vs postpaid; that’s what below image summarizes: response gain vs net extra cost.

Conclusion they report:

  • effects are modest (only a few % points)
  • for this two-part survey, postpaid looks better overall
  • final choice of amount is left to the survey operators (they have to say how much extra \$ per interview they’re willing to pay).

8. Weaknesses they admit

  • Lots of uncertainty on individual coefficients.
  • Choice of which interactions to include was a bit ad hoc.
  • Linear model implies no diminishing returns.
  • Using noninformative priors made it harder to include all interactions—if they had used weakly informative priors to shrink higher-order terms, they could have kept a richer model.

But for illustrating “fit a hierarchical regression → use predictions in a cost–benefit comparison”, it’s good enough.

Core takeaway

This whole section is an example of Bayesian decision analysis without explicitly writing a utility function:

  1. build a hierarchical model from past experiments;
  2. plug in your survey’s characteristics;
  3. predict response lift from each incentive option;
  4. convert to dollars (cost of incentive – calling saved);
  5. hand that curve to the decision-maker.

And that’s exactly why you sometimes keep predictors that are “not significant”: they are needed to make the prediction you actually care about.