Stopping Rules

1. Definition

• A stopping rule is the predefined condition under which an experiment or test is stopped.
• It tells you when to stop collecting data and make a decision.
• Without clear stopping rules, you risk inflating Type I error (false positives) by “peeking” until results look significant.

---

2. Types of Stopping Rules

(a) Fixed-Horizon Stopping Rule (Traditional A/B Test)

• Decide sample size beforehand (e.g., 10,000 users per group).
• Stop once sample is reached.
• One analysis at the end.
• Simple, protects α, but inflexible.

---

(b) Group Sequential Stopping Rules

• Pre-plan interim looks at data (e.g., every 25% of sample).
• Use α-spending rules (like O’Brien–Fleming, Pocock).
• Can stop early if results are overwhelming (efficacy) or hopeless (futility).
• More efficient, widely used in clinical trials.

---

(c) Sequential Probability Ratio Test (SPRT)

• Evaluate likelihood ratio after each data point.
• If ratio exceeds thresholds → stop for H₁ or H₀.
• Otherwise → keep sampling.
• Often requires fewer samples than fixed designs.

---

(d) Bayesian Stopping Rules

• Stop when posterior probability of H₁ (or H₀) exceeds a threshold.
• Example: Stop when $P(H_1 \mid \text{data}) > 0.95$.
• More intuitive (gives probability statements), no α-spending needed.

---

(e) Ethical / Practical Stopping Rules

• In clinical trials:
  • Stop if treatment shows harm (safety).
  • Stop if treatment shows clear benefit (unethical to withhold).
  • Stop if treatment shows futility (unlikely to ever show benefit).

---

3. Examples

A/B Test Example

• Fixed rule: Run until 50,000 visitors per group.
• Sequential rule: Check every 5,000 visitors. Stop early if p < 0.001 (OBF rule).
• Bayesian rule: Stop when posterior probability > 0.95 that B > A.

Clinical Trial Example

• Planned for 1,000 patients.
• Interim checks after 250, 500, 750.
• Stopping rules:
  • If new drug reduces mortality significantly early → stop.
  • If harm observed → stop immediately.
  • If no chance of benefit → stop for futility.

---

4. Why Stopping Rules Matter

• Prevent p-hacking (stopping as soon as results look “good”).
• Ensure correct Type I error control.
• Save time, cost, and resources (stop early if results are clear).
• Protect patients/participants (clinical trials).

---

5. Key Takeaways

• Stopping rules = when and how you end an experiment.
• They can be fixed-horizon (traditional) or sequential/adaptive (group sequential, SPRT, Bayesian).
• Good stopping rules protect against false positives while improving efficiency and ethics.

---

In short:
Stopping rules are pre-specified conditions for when to stop an experiment. Traditional rules use a fixed sample size, while modern sequential and Bayesian rules allow interim looks or continuous monitoring without inflating false positives.