Fixed-Horizon Testing

Fixed-horizon testing means choosing the stopping point before looking at the results, collecting until you reach it, and testing once. The horizon is usually a sample size, sometimes a calendar window. Everything the design promises depends on that order — decide, then collect, then test.

Ordinary fixed-sample analysis in A/B testing assumes this plan. Unplanned stopping based on interim significance can change its false-positive rate, as the simulation below illustrates.

What the horizon costs and what it buys

import numpy as np
from scipy.stats import norm

z_alpha, z_beta = norm.ppf(0.975), norm.ppf(0.80)
p1, p2 = 0.050, 0.055
n = (z_alpha + z_beta) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2
print(f"  detecting 5.0% -> 5.5% at 80% power, two-sided 0.05: {n:,.0f} per variant")
print(f"  halve the effect to 0.25 points and it becomes "
      f"{(z_alpha + z_beta) ** 2 * (p1 * (1 - p1) + 0.0525 * 0.9475) / 0.0025 ** 2:,.0f}")
"""Sample size scales with one over the squared effect. The horizon is a
   commitment made before any data exists, from a number you guessed."""

rng = np.random.default_rng(0)
N = 400_000
steps = rng.normal(size=(N, 20))
final = steps.sum(axis=1) / np.sqrt(20)
print(f"\n  one test at the horizon, no peeking: false positive rate "
      f"{float((np.abs(final) >= 1.96).mean()):.4f}")

cum = np.cumsum(steps, axis=1)
zs = cum / np.sqrt(np.arange(1, 21))
print(f"\n{'unplanned looks':>17}{'false positive rate':>22}{'inflation':>12}")
for k in (1, 2, 4, 10, 20):
    idx = np.linspace(20 / k, 20, k).astype(int) - 1
    rate = float((np.abs(zs[:, idx]) >= 1.96).any(axis=1).mean())
    print(f"{k:>17}{rate:>22.4f}{rate / 0.05:>11.1f}x")
#   detecting 5.0% -> 5.5% at 80% power, two-sided 0.05: 31,231 per variant
#   halve the effect to 0.25 points and it becomes 122,121
#
#   one test at the horizon, no peeking: false positive rate 0.0497
#
#   unplanned looks   false positive rate   inflation
#                 1                0.0497        1.0x
#                 2                0.0829        1.7x
#                 4                0.1262        2.5x
#                10                0.1937        3.9x
#                20                0.2477        5.0x

The first lines are the commitment. Detecting a move from 5.0% to 5.5% at 80% power needs 31,231 users per variant; halve the effect you are willing to miss and it becomes 122,121. Sample size scales with one over the squared effect, so the horizon is fixed by a number you had to guess before any data existed — see statistical power for where that guess comes from.

Under the simulated independent sampling model, one test at the horizon gives a false positive rate of 0.0497, close to the nominal 0.05. This verifies that setting, not every study that follows a calendar or sample-size plan.

The table prices the violation. One extra look moves the rate from 0.0497 to 0.0829 — a single peek costs more than half the error budget again. Four looks reach 0.1262, and twenty reach 0.2477, five times the nominal rate. Nothing in the test statistic changes; the same z, the same 1.96, the same arithmetic. What changed is how many chances it had.

The 0.05 is a target rejection rate over repeated null experiments, not the meaning of a particular p-value. Following the horizon is necessary for this ordinary fixed-sample analysis, but the sampling model, independence and test assumptions must also hold. Valid sequential or adaptive procedures use different calibrations; data-dependent analysis is not invalid by definition.

What is and is not forbidden

The rule is narrower than the folklore. Looking at the data is not the problem — deciding to stop on the basis of an unadjusted interim significance test is. Several things remain perfectly legitimate inside a fixed-horizon design:

• Monitoring for operational faults: broken instrumentation, skewed assignment, a variant that fails to render.
• Stopping for safety or ethics. The experiment ends; the inferential claim afterward may need an adjusted analysis, which is a separate question from whether stopping was right.
• Reporting the running estimate to stakeholders who understand it is not a decision rule.

What is forbidden is exactly the thing the table measures: computing an ordinary p-value at an unplanned moment and letting its value decide whether to continue.

When it is the wrong design

Fixed-horizon testing is inefficient when the true effect is much larger than planned, because the design keeps collecting after the answer is obvious. It is also fragile in the other direction: an effect-size assumption that was too optimistic produces an underpowered study that cannot be rescued after the fact, and one that was too pessimistic buys a sample far larger than needed.

Calendar-based horizons carry an extra wrinkle. “Run for two weeks” leaves the sample size to traffic, so the realized power is unknown until the end, and outcomes that arrive late — returns, cancellations, delayed conversions — may not be observed for the users counted. The analysis plan has to say how those are handled before it can be said to have been followed.

The alternative is not to peek anyway. It is to plan the looks: group sequential designs spend the same 0.05 across several prespecified analyses and permit valid early stopping, and sequential testing covers the family. Choosing one of those is a design decision made at the same moment as the horizon would have been — before the data.

References

• SciPy documentation, scipy.stats.norm. docs.scipy.org
• NumPy documentation, numpy.random.Generator. numpy.org