1. Definition
- A Bayesian stopping rule is a predefined criterion, based on posterior probabilities or Bayes factors, that determines when to stop collecting data in an experiment.
- Instead of controlling Type I error (α) like in frequentist tests, Bayesian stopping rules focus on evidence strength in favor of one hypothesis.
In short: Stop when the posterior probability of a hypothesis is “high enough.”
2. Common Bayesian Stopping Rules
(a) Posterior Probability Thresholds
- Stop if posterior probability of the treatment effect exceeds a threshold.
$P(H_1 \mid \text{data}) > 0.95 \quad \Rightarrow \quad \text{Stop, accept } H_1$
- Stop for futility if:
$P(H_0 \mid \text{data}) > 0.95$
(b) Bayes Factor Thresholds
- Stop when Bayes factor (BF) reaches strong evidence levels.
$BF = \frac{P(\text{data} \mid H_1)}{P(\text{data} \mid H_0)}$
- Decision rules (Jeffreys’ scale):
- BF > 10 → strong evidence for $H_1$.
- BF < 1/10 → strong evidence for $H_0$.
- Otherwise → keep sampling.
(c) Credible Interval Rules
- Stop if the posterior credible interval for the treatment effect excludes zero (or another null value).
- Example: 95% credible interval for difference in conversion rates = [0.01, 0.03] → stop, since effect > 0 with high confidence.
3. Advantages of Bayesian Stopping Rules
- No α-inflation from peeking → Bayesian methods naturally allow continuous monitoring.
- Interpretability → Results can be stated directly:
- “There is a 97% probability that variant B is better than A.”
- Flexibility → Can incorporate prior beliefs.
- Resource efficiency → Stop experiments early if evidence is overwhelming.
4. Disadvantages
- Requires specifying priors, which can influence results.
- More computationally intensive (Monte Carlo, MCMC often needed).
- Stakeholders may be more familiar with p-values than posterior probabilities.
5. Example – Bayesian A/B Test
- Control conversion = 5%.
- Treatment observed = 5.6% after 5,000 users.
- Posterior probability $P(p_B > p_A \mid \text{data}) = 0.97$.
Bayesian stopping rule: If posterior > 0.95 → stop.
Decision: Stop, roll out B.
6. Comparison with Frequentist Stopping Rules
| Feature | Frequentist (Fixed/Sequential) | Bayesian |
|---|---|---|
| Metric | p-value, α, test statistic | Posterior probability, Bayes Factor |
| Peeking | Inflates Type I error (needs correction) | Allowed (no α inflation) |
| Decision | Reject/fail to reject H₀ | Probability statement (“95% chance B is better”) |
| Stopping | Fixed horizon or α-spending | Posterior/BF threshold |
| Interpretability | Harder for non-statisticians | More intuitive |
7. Key Takeaway
- Bayesian stopping rules use posterior probability, Bayes factors, or credible intervals to decide when to stop.
- They allow continuous monitoring without inflating false positives, making them ideal for online experiments and adaptive clinical trials.
In short:
Bayesian stopping rules let you stop an experiment once the posterior probability or Bayes factor shows strong enough evidence for (or against) a treatment effect. Unlike frequentist rules, they tolerate continuous monitoring and provide more intuitive probability statements.