Bayesian Sequential Testing

1. Definition

• Bayesian Sequential Testing is a method where evidence is evaluated continuously (or at interim looks) using Bayesian updating, instead of frequentist p-values and fixed α thresholds.
• At each step, we compute a posterior probability or Bayes factor to decide whether to:
  1. Stop and accept $H_1$​ (treatment effect exists),
  2. Stop and accept $H_0$​ (no effect), or
  3. Continue sampling.

Unlike fixed-horizon tests, Bayesian sequential testing does not require a pre-specified sample size.

---

2. Core Concepts

(a) Bayes’ Theorem

$P(H \mid \text{data}) = \frac{P(\text{data} \mid H) \cdot P(H)}{P(\text{data})}$

• Updates prior belief $P(H)$ with evidence from data.
• Posterior probability = updated belief after seeing data.

(b) Bayes Factor (BF)

$BF = \frac{P(\text{data} \mid H_1)}{P(\text{data} \mid H_0)}$

• BF > 1 → favors $H_1$​.
• BF < 1 → favors $H_0$​.
• Decision thresholds often used:
  • BF > 10 → strong evidence for $H_1$​.
  • BF < 1/10 → strong evidence for $H_0$​.
  • Otherwise → keep sampling.

---

3. Decision Rules in Bayesian Sequential Testing

• Stop for efficacy: Posterior probability $P(H_1 \mid \text{data}) > 0.95$ (or BF > 10).
• Stop for futility: Posterior probability $P(H_0 \mid \text{data}) > 0.95$ (or BF < 1/10).
• Continue: If evidence is inconclusive.

Unlike frequentist tests, Bayesian tests do not inflate Type I error when data is monitored continuously.

---

4. Example – A/B Test

• Testing if version B has higher conversion than version A.
• Prior: $P(H_1) = 0.5, P(H_0) = 0.5$.
• After 5,000 visitors: posterior $P(H_1 \mid data) = 0.93$. → Keep sampling.
• After 8,000 visitors: posterior = 0.97 → stop, conclude B is better.

---

5. Advantages

• No fixed horizon required → you can stop anytime.
• Interpretability: Results expressed as probabilities (e.g., “There’s a 97% probability variant B is better”), instead of abstract p-values.
• Flexibility: Can incorporate prior information.
• Controls error naturally: Doesn’t require α-spending corrections like group sequential tests.

---

6. Disadvantages

• Requires specifying priors (subjective choices can influence results).
• More computationally intensive (especially with complex models).
• Interpretation depends on prior assumptions (which some stakeholders may challenge).

---

7. Comparison with Frequentist Sequential Methods

Feature	Frequentist (SPRT, OBF, Pocock)	Bayesian Sequential
Evidence metric	p-values, likelihood ratios	Posterior probs, Bayes Factors
Stopping rule	Fixed α with α-spending	Flexible thresholds (e.g., BF > 10)
Sample size	Fixed or preplanned groups	Can stop anytime
Multiple looks	Requires correction	Not a problem
Interpretability	Probability of data under H₀	Probability of H₀ or H₁ given data

---

8. Key Takeaway

• Bayesian Sequential Testing evaluates evidence continuously via posterior probabilities or Bayes factors.
• It avoids problems of “peeking” and offers a more intuitive interpretation, but requires priors and more computation.

---

In short:
Bayesian Sequential Testing lets you monitor an experiment continuously and stop when the evidence (posterior probability or Bayes factor) is strong enough. Unlike traditional fixed-horizon tests, it doesn’t inflate Type I error from repeated looks and provides results directly interpretable as probabilities.