What we’re doing with simulation draws

In Bayesian work we often summarize the posterior by random draws:

  • report quantiles of a scalar estimand (2.5%, 25%, 50%, 75%, 97.5%)
  • make scatterplots for 2–3 parameters
  • simulate predictive quantities $\tilde y \sim p(\tilde y \mid \theta_s)$
  • simulate replicated data $y^{\text{rep}}_s$​ for model checking

So the basic setup is: we have independent posterior draws $\theta_s,\quad s = 1,\dots, S$

and we want $S$ big enough that Monte Carlo error doesn’t swamp the actual posterior uncertainty.

Rule-of-thumb result

If $\theta$ is roughly normal with posterior sd $\sigma_\theta$​, and you estimate its mean by the sample mean of $S$ posterior draws, then Monte Carlo error is about $\frac{\sigma_\theta}{\sqrt{S}}.$

They point out that the total variability in the estimate becomes $\sigma_\theta \sqrt{1 + 1/S}.$

For $S = 100$, $1 + 1/100 = 1.01$ (they round to 1.005), so MC error adds almost nothing. That’s why they say about 100 independent draws is often fine for basic posterior summaries.

People often take more (like 1,000) just so printed numbers don’t jiggle around, but it usually doesn’t change the substantive conclusion.

When you need more draws

Some things are estimated more slowly than others.

  1. Posterior probabilities near 0.5
    Standard error is $\sqrt{\frac{p(1-p)}{S}}.$ So with $S=100$ and $p \approx 0.5$, SE ≈ 0.05 → ±5% accuracy.
    Want ±1%? Then you need about $S = 2500$.
  2. Rare events (e.g. $\Pr(\theta_1 > 50 \mid y)$
    If the event is very rare, your 100 or 200 draws might show zero occurrences → only tells you “probability < 1/S”.
    In their 8-schools example: with 200 draws, none exceeded 50; with 10,000 draws, 3 did → estimate 0.0003.
    So: rare events need lots of draws unless you use a more analytic trick.
  3. Extreme quantiles and higher-dimensional summaries
    These are always harder — more draws help.

Nice trick: semi-analytic probabilities

They show a workaround: in the 8-schools model, conditional on $(\mu, \tau)$, the school effect $\theta_1$​ is normal, so $\Pr(\theta_1 > 50 \mid \mu, \tau, y)$

can be computed with a normal CDF. Then you just average those probabilities over 200 posterior draws of $(\mu, \tau)$. That way you don’t need 10,000 draws of $\theta_1$​ itself to estimate a tiny tail probability.

So: sometimes you can keep $S$ small if you push a bit more analytic work into the last step.

The 8-schools illustration

  • With 200 draws they got median 10 and intervals that look sensible.
  • With another 200 draws they got nearly the same thing.
  • With 10,000 draws they got… basically the same thing.
    → So for those central summaries, 200 was already good enough.

For a probability like $\Pr(\theta_1 > \theta_3 \mid y)$, with 200 draws they got 0.73, and the SE is $\sqrt{0.73(1-0.73)/200} \approx 0.03,$

which is perfectly fine.

Bottom line

  • Central stuff (means, medians, 50% and 95% intervals, probabilities near 0.5): ~100–200 independent draws is usually ok.
  • More precise probabilities (±1%): a few thousand.
  • Very rare events / extreme tails: many more draws, or use an analytic/conditional trick.
  • That’s why in most applied examples they use 100 to 2000 draws — you usually don’t need more to say something sensible.