Sequential Testing

Sequential testing is a design that analyzes data as it accumulates and allows the experiment to stop early, while keeping the false-positive rate at the level it claims. It is the alternative to a fixed-horizon test, where the sample size is set in advance and the result is read once.

The reason it exists is that people look anyway. Watching a running experiment and stopping when the p-value first dips below 0.05 is not a 5 percent procedure, and the gap is not small.

What repeated looks cost

Each look is another chance for noise to cross the line, and the chances accumulate. The calculation below simulates a test statistic under the null hypothesis — no effect at all — and counts how often it crosses the ordinary 1.96 boundary at least once, across equally spaced looks.

import numpy as np
from scipy import stats

def crossing_rate(boundaries, replicates=200_000, seed=5):
    """How often a z statistic under the null crosses any of the looks' boundaries.
    Looks are equally spaced, so the running statistic is a scaled Brownian path."""
    rng = np.random.default_rng(seed)
    k = len(boundaries)
    increments = rng.standard_normal((replicates, k))
    partial = np.cumsum(increments, axis=1)
    z = partial / np.sqrt(np.arange(1, k + 1))
    return float((np.abs(z) >= np.array(boundaries)).any(axis=1).mean())

fixed = stats.norm.ppf(1 - 0.05 / 2)
for k in (1, 2, 5, 10, 20):
    print(f"{k:>2} {crossing_rate([fixed] * k):.4f}")
# 1 0.0494
# 2 0.0823
# 5 0.1413
# 10 0.1923
# 20 0.2488

pocock = 2.413
# Published boundaries for five equally spaced looks at a two-sided 5 percent level.
obrien_fleming = [4.562, 3.226, 2.634, 2.281, 2.040]
print(f"{crossing_rate([pocock] * 5):.4f} {crossing_rate(obrien_fleming):.4f}")
# 0.0505 0.0495

A daily check on a three-week experiment is twenty looks, and the procedure it produces rejects a true null about a quarter of the time. Kohavi and colleagues make the general point about repeated comparisons: without adjustment “there is an increased risk of finding what appear to be significant results by chance”. The mechanism is covered under risk of peeking; sequential testing is the constructive answer to it.

Spending the error budget

The idea behind a group sequential design is that the 5 percent is a budget for the whole experiment rather than a level available afresh at each look. Interim analyses are planned in advance, and each is given a boundary stricter than 1.96 so that the total chance of crossing at any of them comes back to 5 percent.

Two classical schemes divide the budget differently, and the choice is a statement about when you want to be able to stop.

Both hold the overall rate: in the same simulation, the Pocock boundary crosses 5.1 percent of the time under the null across five looks and the O’Brien–Fleming boundaries 5.0 percent. The trade is visible in the last row of the table. Pocock buys early stopping by making the final verdict harder, which is why a test that runs to completion under Pocock can fail to reject a result a fixed-horizon test would have called significant. O’Brien–Fleming gives up almost all early stopping and ends where a fixed design ends.

Both schemes require the number and timing of looks to be fixed in advance, which is awkward when an experiment’s traffic is unpredictable. Lan and DeMets removed that constraint with the alpha spending function: instead of fixing the looks, the design fixes how much of the budget has been spent by any point in the experiment, and boundaries are computed as analyses actually occur.

The other two families

• The sequential probability ratio test. Wald’s original procedure updates the likelihood ratio between two simple hypotheses after every observation and stops when it crosses either of two thresholds. It is the most sample-efficient option when the two hypotheses are genuinely simple and specified in advance, which in practice they rarely are.
• Bayesian monitoring. Update the posterior as data arrive and stop when the probability of a worthwhile effect crosses a pre-committed level — see posterior probability of uplift. The framework changes what the number means, and it does not by itself exempt an optional stopping rule chosen to favor a result. Methods described as always-valid are built so that the guarantee holds at any stopping time; ordinary posterior monitoring with an ad hoc rule is not the same thing.

What early stopping costs

Stopping early is not free, and two consequences are routinely missed.

The unadjusted effect estimate from a trial stopped early for success tends to be biased upward. Crossing a boundary requires the observed effect to be large, and among the paths that cross early a disproportionate share are the ones where noise happened to help. A campaign sized on that estimate risks underdelivering, so the point estimate from an early stop should be read with that optimism in mind rather than taken as the best guess; it is not a guaranteed upper bound on the true effect. Estimators adjusted for the stopping rule exist for exactly this, and so does confirming the effect in an independent follow-up.

And failing to cross a boundary is not a finding of no effect. It means the evidence did not reach the pre-set bar within the horizon, which is compatible with a real effect the design was too small to catch. Sequential designs often add a futility boundary specifically so that stopping for “this is not going to work” is a planned decision with its own error rate, rather than an inference read off a failure to reject.

Where the alternative is a team that will look daily regardless, a sequential design is straightforwardly better than a fixed-horizon one with an honor system. Where the experiment can genuinely be left alone, the fixed design is simpler, needs no boundary table, and gives the tightest final answer — see A/B testing and sample size.

References

• Stuart J. Pocock, “Group sequential methods in the design and analysis of clinical trials”, Biometrika 64(2), 191–199, 1977
• Peter C. O’Brien and Thomas R. Fleming, “A multiple testing procedure for clinical trials”, Biometrics 35(3), 549–556, 1979
• K. K. Gordon Lan and David L. DeMets, “Discrete sequential boundaries for clinical trials”, Biometrika 70(3), 659–663, 1983
• Abraham Wald, “Sequential Tests of Statistical Hypotheses”, Annals of Mathematical Statistics 16(2), 117–186, 1945
• Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne, “Controlled experiments on the web: survey and practical guide”, Data Mining and Knowledge Discovery 18(1), 2009