1. Definition
- A Traditional A/B Test is the classical, fixed-horizon approach to experimentation.
- You predefine a sample size or test duration, collect data until that point, and then perform the hypothesis test once at the end.
- It is the same as Fixed-Horizon A/B Testing.
2. Process
- Define hypotheses (e.g., H₀: conversion rate A = conversion rate B).
- Choose significance level (α, typically 0.05).
- Run an a priori power analysis to decide the required sample size (n).
- Fix the horizon → e.g., “we will stop after 10,000 users per variant.”
- Collect data until that horizon.
- Perform a statistical test (e.g., two-proportion z-test).
- Reject or fail to reject H₀.
3. Key Characteristics
- Fixed sample size or time window.
- No peeking at interim results → prevents inflated Type I error.
- Simple to implement and widely accepted in classical statistics.
4. Example
- You want to test if a new button color increases conversions.
- Baseline conversion = 5%.
- Minimum Detectable Lift (MDL) = +10%.
- A priori power analysis says you need ~8,000 users per group.
- You run the test until each variant has 8,000 users.
- At the end, you run a two-proportion z-test.
- Only then do you decide if B beats A.
5. Advantages
- Statistically rigorous and widely understood.
- Guarantees correct Type I error control (if assumptions are followed).
- Easy to explain to non-technical stakeholders.
6. Limitations
- Inflexible (must wait until horizon is reached).
- May waste resources if one variant is clearly superior early on.
- Not adaptive to real-time learning.
7. Comparison with Modern Approaches
| Approach | Decision Rule | Flexibility | Efficiency |
|---|---|---|---|
| Traditional A/B (Fixed-Horizon) | Analyze once at end | Rigid | May waste samples |
| Sequential Testing | Allows interim looks, uses α-spending | More flexible | Can stop early |
| Bayesian A/B | Uses probability of superiority | Very flexible | Intuitive interpretation |
| Multi-Armed Bandit | Dynamically allocates traffic | Adaptive | Very efficient |
In short:
A Traditional A/B Test = Fixed-Horizon A/B Test. You set the sample size/duration in advance, collect data until that point, and analyze only once at the end. It’s simple and statistically clean, but less flexible than modern sequential or adaptive approaches.