Multiple Comparisons

The multiple comparisons problem is that testing many things at once makes surprising results ordinary. Each individual test tolerates some chance of a false alarm; run enough tests and at least one false alarm becomes the expected outcome rather than a surprise.

The arithmetic is simple enough to do in your head. With a 5% threshold and twenty independent comparisons, the probability that at least one comes out significant by chance alone is 1 − 0.95²⁰, or about 64%. Real business comparisons are rarely independent, which changes the number but not the conclusion: a single significant result found among many is weak evidence, and it is weak for reasons unrelated to how carefully the analysis was done.

Two organizational habits that create it

• Slicing until something appears. A dashboard that breaks a metric down by twelve dimensions supports hundreds of comparisons, and the differences that get escalated are the extreme ones by construction. Nobody ran a test; the effect is the same.
• Checking a running experiment repeatedly. This is the same problem along the time axis. Evaluating daily and stopping when the result looks good makes the stopping rule part of the analysis, and a test watched continuously will eventually cross a threshold whether or not an effect exists.

Both are ordinary, well-intentioned behaviour. The second is more dangerous because it feels like diligence, and because the resulting number is reported without any mention of how many times it was looked at.

What the ASA statement says about this

The American Statistical Association’s statement on p-values addresses the problem directly in one of its six principles: “Proper inference requires full reporting and transparency.” Reporting one comparison out of forty, without saying so, breaks the reasoning that makes the number mean anything.

Two other principles matter here because they remove the escape routes people reach for. A significant result does not become reliable by clearing a line — “Scientific conclusions and business or policy decisions should not be based only on whether a P-value passes a specific threshold” — and it does not tell you the probability that your explanation is right: “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.”

It helps to recall what a p-value actually is, in the statement’s own words: “the probability under a specified statistical model that a statistical summary of the data … would be equal to or more extreme than its observed value.” It is computed as if this one comparison were the only one — which is exactly the assumption that repeated slicing violates.

Three responses, for different situations

The trade-off in the second row is worth stating plainly: demanding more evidence per comparison reduces power, so a strict adjustment across many segments can hide effects that are real. That is a reason to limit the number of planned comparisons rather than to compensate for an unlimited number.

Do not ban exploration — label it

Exploratory slicing is how most good hypotheses are found, and forbidding it would cost more than the problem it solves. What is needed is a distinction visible in the report: this analysis was planned, that one was found while looking.

Three lines make that practical. Say how many comparisons were examined. Say which were decided in advance. And for a discovered finding, say what would confirm it, so the conversation moves toward a test rather than toward a decision. A finding presented as “we noticed this while exploring and have not confirmed it” survives scrutiny; the same finding presented as a result does not, and loses credibility for the analysis around it.

For experiments specifically, the mechanism belongs in the design: fixing the metric, segments, and run length in advance is part of what experiment design is for, and deciding those after seeing the data is the main way a well-run test produces a wrong answer. Where interim checks are genuinely needed — to stop a harmful change early — the sensible form is a pre-agreed rule for when to stop, not a daily look with an informal threshold.

How this fits with sampling, bias, and the difference between statistical and practical significance is worked through in Reading a Result Honestly.

Reference: Wasserstein & Lazar, The ASA’s Statement on p-Values: Context, Process, and Purpose, The American Statistician 70:129–133, 2016 (principles as reproduced in Table I of this open-access discussion).