1. Definition
- Bayesian Sequential Testing is a method where evidence is evaluated continuously (or at interim looks) using Bayesian updating, instead of frequentist p-values and fixed α thresholds.
- At each step, we compute a posterior probability or Bayes factor to decide whether to:
- Stop and accept $H_1$ (treatment effect exists),
- Stop and accept $H_0$ (no effect), or
- Continue sampling.
Unlike fixed-horizon tests, Bayesian sequential testing does not require a pre-specified sample size.
2. Core Concepts
(a) Bayes’ Theorem
$P(H \mid \text{data}) = \frac{P(\text{data} \mid H) \cdot P(H)}{P(\text{data})}$
- Updates prior belief $P(H)$ with evidence from data.
- Posterior probability = updated belief after seeing data.
(b) Bayes Factor (BF)
$BF = \frac{P(\text{data} \mid H_1)}{P(\text{data} \mid H_0)}$
- BF > 1 → favors $H_1$.
- BF < 1 → favors $H_0$.
- Decision thresholds often used:
- BF > 10 → strong evidence for $H_1$.
- BF < 1/10 → strong evidence for $H_0$.
- Otherwise → keep sampling.
3. Decision Rules in Bayesian Sequential Testing
- Stop for efficacy: Posterior probability $P(H_1 \mid \text{data}) > 0.95$ (or BF > 10).
- Stop for futility: Posterior probability $P(H_0 \mid \text{data}) > 0.95$ (or BF < 1/10).
- Continue: If evidence is inconclusive.
Unlike frequentist tests, Bayesian tests do not inflate Type I error when data is monitored continuously.
4. Example – A/B Test
- Testing if version B has higher conversion than version A.
- Prior: $P(H_1) = 0.5, P(H_0) = 0.5$.
- After 5,000 visitors: posterior $P(H_1 \mid data) = 0.93$. → Keep sampling.
- After 8,000 visitors: posterior = 0.97 → stop, conclude B is better.
5. Advantages
- No fixed horizon required → you can stop anytime.
- Interpretability: Results expressed as probabilities (e.g., “There’s a 97% probability variant B is better”), instead of abstract p-values.
- Flexibility: Can incorporate prior information.
- Controls error naturally: Doesn’t require α-spending corrections like group sequential tests.
6. Disadvantages
- Requires specifying priors (subjective choices can influence results).
- More computationally intensive (especially with complex models).
- Interpretation depends on prior assumptions (which some stakeholders may challenge).
7. Comparison with Frequentist Sequential Methods
| Feature | Frequentist (SPRT, OBF, Pocock) | Bayesian Sequential |
|---|---|---|
| Evidence metric | p-values, likelihood ratios | Posterior probs, Bayes Factors |
| Stopping rule | Fixed α with α-spending | Flexible thresholds (e.g., BF > 10) |
| Sample size | Fixed or preplanned groups | Can stop anytime |
| Multiple looks | Requires correction | Not a problem |
| Interpretability | Probability of data under H₀ | Probability of H₀ or H₁ given data |
8. Key Takeaway
- Bayesian Sequential Testing evaluates evidence continuously via posterior probabilities or Bayes factors.
- It avoids problems of “peeking” and offers a more intuitive interpretation, but requires priors and more computation.
In short:
Bayesian Sequential Testing lets you monitor an experiment continuously and stop when the evidence (posterior probability or Bayes factor) is strong enough. Unlike traditional fixed-horizon tests, it doesn’t inflate Type I error from repeated looks and provides results directly interpretable as probabilities.