Traditional A/B Test (Fixed-Horizon A/B Test)

1. Definition

• A Traditional A/B Test is the classical, fixed-horizon approach to experimentation.
• You predefine a sample size or test duration, collect data until that point, and then perform the hypothesis test once at the end.
• It is the same as Fixed-Horizon A/B Testing.

---

2. Process

1. Define hypotheses (e.g., H₀: conversion rate A = conversion rate B).
2. Choose significance level (α, typically 0.05).
3. Run an a priori power analysis to decide the required sample size (n).
4. Fix the horizon → e.g., “we will stop after 10,000 users per variant.”
5. Collect data until that horizon.
6. Perform a statistical test (e.g., two-proportion z-test).
7. Reject or fail to reject H₀.

---

3. Key Characteristics

• Fixed sample size or time window.
• No peeking at interim results → prevents inflated Type I error.
• Simple to implement and widely accepted in classical statistics.

---

4. Example

• You want to test if a new button color increases conversions.
• Baseline conversion = 5%.
• Minimum Detectable Lift (MDL) = +10%.
• A priori power analysis says you need ~8,000 users per group.
• You run the test until each variant has 8,000 users.
• At the end, you run a two-proportion z-test.
• Only then do you decide if B beats A.

---

5. Advantages

• Statistically rigorous and widely understood.
• Guarantees correct Type I error control (if assumptions are followed).
• Easy to explain to non-technical stakeholders.

---

6. Limitations

• Inflexible (must wait until horizon is reached).
• May waste resources if one variant is clearly superior early on.
• Not adaptive to real-time learning.

---

7. Comparison with Modern Approaches

Approach	Decision Rule	Flexibility	Efficiency
Traditional A/B (Fixed-Horizon)	Analyze once at end	Rigid	May waste samples
Sequential Testing	Allows interim looks, uses α-spending	More flexible	Can stop early
Bayesian A/B	Uses probability of superiority	Very flexible	Intuitive interpretation
Multi-Armed Bandit	Dynamically allocates traffic	Adaptive	Very efficient

---

In short:
A Traditional A/B Test = Fixed-Horizon A/B Test. You set the sample size/duration in advance, collect data until that point, and analyze only once at the end. It’s simple and statistically clean, but less flexible than modern sequential or adaptive approaches.