Risk of Peeking

Peeking is checking an experiment while it runs and stopping as soon as the result looks significant. The problem arises when the analysis uses fixed-sample inference even though the result determined the stopping time. Its error guarantee was derived for a planned final analysis; valid sequential procedures instead account for repeated looks and stopping.

Stopping at the first crossing of an unadjusted fixed-sample threshold changes the procedure whose error rate you report.

Why repeated looks inflate the false-positive rate

A test statistic on accumulating data wanders. Even with no real effect it will drift up and down, and each additional look is another opportunity for it to be past the threshold at that moment. Stopping at the first such moment selects for exactly those excursions.

A rough sense of the scale: if the looks were independent, \(k\) of them at \(\alpha = 0.05\) would give a probability of at least one false positive of \(1-(1-\alpha)^k\) — about 0.23 at five looks and 0.40 at ten. Successive looks at accumulating data are dependent. The independent calculation is an illustration, not a general bound: evaluate error under the actual test, joint distribution, monitoring schedule, and stopping rule. See Type I error.

This is the same family of problem as running many comparisons, and the ASA names the general practice in its discussion — “‘p-hacking‘ and ‘data dredging’ that emphasize the search for small p-values over other statistical and scientific reasoning” — with the remedy stated as a principle: “proper inference requires full reporting and transparency.“

How it happens without anyone deciding to

• A dashboard that shows significance live. Daily viewing becomes an inference problem when it changes stopping, selection, or the final claim without a method that accounts for those choices. Monitoring alone can retain a preplanned final analysis.
• A weekly review that asks “can we call it yet?” The question itself is the stopping rule.
• Ending a test early because it looks bad. Symmetric to the optimistic version and equally distorting.
• Extending a test because it is “almost” significant. The mirror image — a stopping rule that depends on the result, applied in the other direction.

The last two are worth dwelling on, because neither feels like cheating. Both make the duration a function of the data, which is precisely what the fixed-horizon calculation assumed away.

Four legitimate responses

1. Fix the horizon in advance and test once. Compute the sample size, run to it, then look — see power analysis and sample size. Simple and often the right answer.
2. Use a method designed for repeated looks. Sequential designs spend the error budget across planned interim analyses — see sequential testing and stopping rules.
3. Look, but only for safety. Monitoring for a serious regression and being willing to abort is a different decision from calling a winner, and the guardrail check does not have to share the primary metric’s error budget — as long as an abort is not reported as a result.
4. Report an interval rather than a verdict. A fixed-sample interval does not fix outcome-dependent stopping, and its realized width need not reveal the problem. Report the effect size, but use a sequentially valid interval or confidence sequence when the stopping rule requires one. See confidence intervals.

The distinction that makes all of this usable: monitoring and outcome-dependent stopping are different. Keep the planned final analysis, or use a sequential design or anytime-valid method whose error guarantees cover the actual looks and stopping decisions. Specify the procedure in advance, but remember that simply writing down “stop at the first fixed-sample p < .05” does not control the overall error rate. See hypothesis testing and A/B testing.

Further reading

• NIST/SEMATECH e-Handbook of Statistical Methods, §7.1.3 “What are statistical tests?”
• American Statistical Association, statement on statistical significance and p-values

For inference valid across monitoring times, see Howard et al., Time-uniform, nonparametric, nonasymptotic confidence sequences.