1. Definition
- Fixed-horizon testing is a statistical testing approach where you:
- Decide in advance the sample size (n) or the end time of the test.
- Collect data until that point.
- Run the hypothesis test once at the end.
The “horizon” refers to the predefined stopping point (fixed sample size or time).
2. Why It’s Important
- Protects the validity of Type I error rate (α).
- Prevents peeking (checking results early and stopping when significant → inflates false positive rate).
- Ensures that p-values and confidence intervals remain valid under the assumptions.
3. Process
- Define hypotheses (H₀ and H₁).
- Choose significance level (α, e.g., 0.05).
- Perform a priori power analysis to determine required sample size.
- Fix the horizon (sample size n or duration).
- Collect data until that point.
- Conduct test once → make decision.
4. Example – A/B Test
- Goal: Detect a +10% lift in conversion (baseline 5% → 5.5%).
- α = 0.05, Power = 0.80.
- A priori power analysis → need ≈ 7,850 users per variant.
- Horizon fixed: stop after 7,850 users per group.
- At the end: run two-proportion z-test once.
If p ≤ 0.05 → reject H₀. If not → fail to reject H₀.
5. Advantages
- Simple and widely accepted.
- Preserves Type I error guarantees.
- Easy to explain and implement.
6. Limitations
- Inflexible: can’t stop early even if results are obvious.
- May waste resources if one variant is clearly better (or worse).
- Doesn’t allow continuous monitoring of results.
7. Comparison with Sequential Testing
| Feature | Fixed-Horizon Testing | Sequential / Adaptive Testing |
|---|---|---|
| Stop rule | Predefined sample size/time | Can stop early based on data |
| Error rate (α) | Controlled at chosen level | Must use special methods (e.g., alpha spending) |
| Efficiency | May waste samples if effects are strong | More efficient, can stop earlier |
| Practical use | Traditional experiments, clinical trials, classic A/B tests | Online experiments, bandits, adaptive trials |
In short:
Fixed-horizon testing means you predefine a stopping point (sample size or time), run the experiment until then, and analyze results only once at the end. It’s simple, preserves error rates, but less flexible than sequential/adaptive testing.