Bayesian Stopping Rules

1. Definition

• A Bayesian stopping rule is a predefined criterion, based on posterior probabilities or Bayes factors, that determines when to stop collecting data in an experiment.
• Instead of controlling Type I error (α) like in frequentist tests, Bayesian stopping rules focus on evidence strength in favor of one hypothesis.

In short: Stop when the posterior probability of a hypothesis is “high enough.”

---

2. Common Bayesian Stopping Rules

(a) Posterior Probability Thresholds

• Stop if posterior probability of the treatment effect exceeds a threshold.

$P(H_1 \mid \text{data}) > 0.95 \quad \Rightarrow \quad \text{Stop, accept } H_1$

• Stop for futility if:

$P(H_0 \mid \text{data}) > 0.95$

(b) Bayes Factor Thresholds

• Stop when Bayes factor (BF) reaches strong evidence levels.

$BF = \frac{P(\text{data} \mid H_1)}{P(\text{data} \mid H_0)}$

• Decision rules (Jeffreys’ scale):
  • BF > 10 → strong evidence for $H_1$​.
  • BF < 1/10 → strong evidence for $H_0$​.
  • Otherwise → keep sampling.

(c) Credible Interval Rules

• Stop if the posterior credible interval for the treatment effect excludes zero (or another null value).
• Example: 95% credible interval for difference in conversion rates = [0.01, 0.03] → stop, since effect > 0 with high confidence.

---

3. Advantages of Bayesian Stopping Rules

• No α-inflation from peeking → Bayesian methods naturally allow continuous monitoring.
• Interpretability → Results can be stated directly:
  • “There is a 97% probability that variant B is better than A.”
• Flexibility → Can incorporate prior beliefs.
• Resource efficiency → Stop experiments early if evidence is overwhelming.

---

4. Disadvantages

• Requires specifying priors, which can influence results.
• More computationally intensive (Monte Carlo, MCMC often needed).
• Stakeholders may be more familiar with p-values than posterior probabilities.

---

5. Example – Bayesian A/B Test

• Control conversion = 5%.
• Treatment observed = 5.6% after 5,000 users.
• Posterior probability $P(p_B > p_A \mid \text{data}) = 0.97$.

Bayesian stopping rule: If posterior > 0.95 → stop.
Decision: Stop, roll out B.

---

6. Comparison with Frequentist Stopping Rules

Feature	Frequentist (Fixed/Sequential)	Bayesian
Metric	p-value, α, test statistic	Posterior probability, Bayes Factor
Peeking	Inflates Type I error (needs correction)	Allowed (no α inflation)
Decision	Reject/fail to reject H₀	Probability statement (“95% chance B is better”)
Stopping	Fixed horizon or α-spending	Posterior/BF threshold
Interpretability	Harder for non-statisticians	More intuitive

---

7. Key Takeaway

• Bayesian stopping rules use posterior probability, Bayes factors, or credible intervals to decide when to stop.
• They allow continuous monitoring without inflating false positives, making them ideal for online experiments and adaptive clinical trials.

---

In short:
Bayesian stopping rules let you stop an experiment once the posterior probability or Bayes factor shows strong enough evidence for (or against) a treatment effect. Unlike frequentist rules, they tolerate continuous monitoring and provide more intuitive probability statements.