Fixed-Horizon Testing

1. Definition

• Fixed-horizon testing is a statistical testing approach in which you:
  1. Choose the sample size, analysis time, or another stopping rule before examining the results.
  2. Collect data until the predefined stopping point is reached.
  3. Perform the planned hypothesis test at the end.

The “horizon” is the predefined stopping point, such as a fixed number of observations or a fixed calendar duration.

If a calendar duration is used, the final sample size may vary with traffic. The analysis plan should specify how eligibility, exposure, late-arriving outcomes, and incomplete observations will be handled.

---

2. Why It’s Important

• Controls the Type I error rate at the selected significance level when the test assumptions and analysis plan are followed.
• Prevents unadjusted optional stopping, where researchers repeatedly inspect significance and stop when a favorable result appears.
• Keeps conventional p-values and confidence intervals valid under their stated assumptions.
• Encourages researchers to define the hypothesis, outcome, sample size, and analysis method before observing the results.

Data may be monitored for operational quality or safety, but repeatedly testing the treatment effect and stopping based on an ordinary p-value can inflate the false-positive rate.

---

3. Process

1. Define the null and alternative hypotheses.
2. Choose the primary outcome and statistical test.
3. Select the significance level, such as $\alpha = 0.05$.
4. Specify the minimum effect size worth detecting.
5. Perform an a priori power analysis to estimate the required sample size.
6. Define the stopping rule, allocation ratio, exclusions, and analysis population.
7. Collect data until the stopping rule is reached.
8. Conduct the planned analysis and report the effect estimate, confidence interval, and p-value.

---

4. Example – A/B Test

• Goal: Detect a 10% relative lift in conversion, from 5.0% to 5.5%.
• Absolute effect: 0.5 percentage points.
• Two-sided significance level: $\alpha = 0.05$.
• Statistical power: 80%.
• Equal allocation between the two variants.
• A standard normal approximation requires approximately 31,300 users per variant.
• The experiment stops after the planned sample size is reached and the required outcomes are observed.
• A two-proportion z-test is performed once as planned.

If the resulting p-value is at or below 0.05, reject $H_0$. Otherwise, fail to reject $H_0$.

The exact sample-size estimate may vary slightly with the power-analysis method, continuity correction, expected attrition, allocation ratio, or statistical test.

---

5. Advantages

• Simple and widely understood.
• Controls the planned Type I error rate when implemented correctly.
• Easy to document, explain, and reproduce.
• Works well when the expected effect and required sample size can be estimated reliably.
• Avoids the additional design complexity of sequential stopping boundaries.

---

6. Limitations

• Ordinary fixed-horizon inference does not support stopping early because the interim result looks statistically significant.
• Stopping early for operational, ethical, or safety reasons is possible, but inferential claims may require an adjusted analysis.
• The experiment may use more observations than a well-designed sequential test when the true effect is large.
• An incorrect effect-size assumption can produce an underpowered experiment or an unnecessarily large sample.
• Fixed-duration designs may end with an unpredictable sample size when traffic or event rates vary.
• Repeated unadjusted significance testing before the horizon invalidates the nominal false-positive guarantee.

---

7. Comparison with Sequential Testing

Feature	Fixed-Horizon Testing	Sequential Testing
Stopping rule	Defined before the experiment and not based on unadjusted interim significance	Uses predefined data-dependent stopping boundaries
Analysis timing	Primary analysis at the planned horizon	Allows valid interim analyses
Type I error	Controlled by the conventional test when the plan is followed	Requires a valid sequential method, such as group-sequential boundaries, alpha spending, or always-valid inference
Sample efficiency	Uses the planned sample unless another prespecified rule applies	Can reduce expected sample size in some scenarios, but is not always more efficient
Complexity	Relatively simple	Requires more design and analysis discipline
Typical use	Traditional experiments and standard A/B tests	Group-sequential trials and online experiments requiring valid early stopping

---

In short:
Fixed-horizon testing uses a stopping point and analysis plan defined before the results are examined. The planned hypothesis test is performed at the horizon, preserving conventional error-rate guarantees when its assumptions are satisfied. It is straightforward but less flexible than a properly designed sequential approach.