Stopping Rules

A stopping rule is the condition under which an experiment stops collecting data and a decision is made, and the defining property of a good one is that the analysis method’s guarantees cover the way the experiment will actually be stopped. The reason is that the error guarantees of a test are guarantees about a procedure, and the procedure includes when you stop: a fixed-sample p-value is valid for one analysis at a predetermined size, and applying it to data that were stopped on the strength of interim looks is a different procedure with a different error rate. Evan Miller’s widely read note on A/B testing states the assumption most dashboards violate: “the significance calculation makes a critical assumption that you have probably violated without even realizing it: that the sample size was fixed in advance”. If instead the experiment is run until a difference looks significant, “all the reported significance levels become meaningless”. The mechanism is described under risk of peeking: a test statistic on accumulating data wanders, and stopping at the first moment it crosses a threshold selects exactly the excursions that a fixed-sample p-value was not designed to count.

So the question is not whether to have a stopping rule, since every experiment stops somehow, but which kind, and what each kind guarantees. The kinds in common use differ in what they fix in advance.

Kinds of rule, and what each fixes in advance

The fixed-horizon rule is the traditional A/B test: compute the sample size from a power analysis, run to it, then test once. It is the simplest rule and often the right one. Its guarantee is conditional: the test’s assumptions have to hold, the metric and the analysis have to be fixed in advance, and the data must not be stopped or extended on the strength of interim results. Waiting to the end is necessary for the guarantee, not sufficient for it.

Group sequential designs are the clinical-trial answer to wanting to look before the end. The US regulator’s guidance on adaptive designs describes them as designs with planned interim analyses at which the trial may stop early, either because the evidence of efficacy is already sufficient or because the trial has become unlikely to show an effect, which is called stopping for futility. The problem they solve is the one above: testing at the conventional level at every interim look “would inflate the Type I error probability”, so the boundaries at each look are set so that the error probability over all the looks together stays at the planned level. The methods differ in how demanding the early looks are. An O’Brien-Fleming boundary requires very persuasive early results before the trial stops for efficacy; a Pocock boundary requires less and stops early more often; an alpha-spending function such as Lan-DeMets specifies how the error probability is spent as the trial progresses. What they share is that they “rely on prospective planning of both the number of interim analyses and the specific sample size or number of event targets at which those analyses will occur.” The sequential probability ratio test is the older relative of these designs, comparing a likelihood ratio against two thresholds after each observation.

Anytime-valid methods remove the need to plan the looks. A confidence sequence, in the definition of the paper that developed modern nonparametric versions, is “a sequence of confidence intervals that is uniformly valid over an unbounded time horizon”: whenever you look, and however often, the interval’s coverage holds, so any stopping rule can be used with it, under the method’s own assumptions about the data. The price is width: at any given sample size the interval is wider than the fixed-sample interval on the same data, because it is paying for the freedom to stop at any time.

Bayesian rules stop when a posterior probability crosses a threshold, for instance when the probability that the variant is better exceeds a stated value. That is a coherent rule and a different question; it does not by itself say how often such a rule stops on a null effect, and a team that also cares about that frequency has to compute it for the rule as specified, prior and threshold included.

Efficacy, futility, and harm are different stops

The guidance treats stopping for efficacy, for futility, and for harm as different decisions, and they carry different statistical and practical weight. Stopping for efficacy is the one that inflates error if done loosely, so the guidance asks that a trial be stopped for efficacy only when the prospectively planned criteria are met. It also has a cost the enthusiasm of an early win hides: “a trial terminated early for efficacy will have a smaller sample size for the evaluation of safety and potentially important secondary efficacy endpoints”, which is why the guidance reserves early efficacy stops for compelling ethical reasons or for results that are highly persuasive in both size and strength of evidence. Stopping for futility, ending a study that is unlikely to show an effect, is treated more leniently: nonbinding futility rules are common, and adding such a rule “does not increase the Type I error probability and is often appropriate”. Stopping for harm is a safety decision, made on the monitored safety data, and it does not need to share the primary metric’s error budget as long as an abort is not later reported as a result.

That last distinction transfers directly to product experiments. Watching a guardrail metric daily and aborting if it deteriorates is monitoring, and it is compatible with a fixed-horizon primary analysis. Watching the primary metric daily and calling a winner when it looks good is a stopping rule, and it changes the error rate of the procedure whatever the final p-value says.

What a reviewer should ask

• What was the stopping rule, and where was it written down before the experiment started?
• If the experiment was looked at before the planned end, was the analysis method one whose guarantees cover those looks?
• If it was stopped early for a positive result, what was the boundary at that look, and is the reported estimate adjusted for having stopped there?
• If it was extended because the result was almost significant, that is a stopping rule applied in the other direction; was it planned?
• Were safety monitoring and the primary decision kept separate?

The whole subject reduces to one principle: the rule for stopping is part of the procedure whose error rate you report, so the analysis method has to be one whose guarantees cover that rule. Agreeing the rule before the data is how a team makes sure that is true and can show it afterward; the agreement is the operational side, and the guarantee comes from matching the method to the rule. How the decision to test at all is set up is covered under hypothesis testing and A/B testing.

References

• Evan Miller, How Not To Run an A/B Test (2010)
• US Food and Drug Administration, Adaptive Designs for Clinical Trials of Drugs and Biologics: Guidance for Industry (November 2019), sections on group sequential designs
• Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon, Time-uniform, nonparametric, nonasymptotic confidence sequences, Annals of Statistics 49(2), 2021