Practical Significance

Practical significance asks whether an effect is large enough to matter — to change a decision, justify a cost, or improve an outcome someone cares about. It is a business judgment about magnitude, made against a threshold, and it is separate from whether the effect is statistically distinguishable from zero.

The American Statistical Association draws the boundary explicitly in one of its six principles on p-values: “A P-value, or statistical significance, does not measure the size of an effect or the importance of the result.” That is not a criticism of the tool; it is a statement about what question it answers.

Where the threshold comes from

The threshold is the smallest effect that would change what you do, and it has to come from the business rather than the data. Three kinds of reasoning produce one.

• Break-even. The change costs a certain amount to build and run. The effect that pays that back is a floor, computable before any measurement.
• Opportunity cost. The team could do something else instead. An effect smaller than the expected value of the alternative is a reason to do the alternative, even if it is real.
• Ongoing burden. Some improvements add permanent complexity — a new code path, an extra integration, a model to maintain. A tiny gain bought with lasting maintenance is frequently a bad trade, and this is the consideration most often omitted.

Write the threshold down before the result arrives. Deciding afterward what counts as meaningful is how a 0.3% lift becomes “a solid improvement” on Monday and “within noise” on Thursday, depending on who is arguing.

Compare the interval with the threshold, not the point estimate

The temptation is to cross the significance verdict with “is the estimate above the threshold” and read off an action. That grid gives the wrong answer whenever the uncertainty is wide relative to the threshold, which at realistic sample sizes is most of the time. What decides the case is where the whole interval sits.

Where the interval sits	What it supports
Entirely above the threshold	Act. The effect is large enough to be worth having even at the unfavorable end of the range
Entirely below the threshold	Decline — including when the result is statistically significant. A real effect too small to justify the change is still a no, and this is the case people find hardest to accept
Straddling the threshold	Undecided. The measurement does not settle the decision, whichever side of significance it fell on. Say so, rather than reporting the midpoint and letting the reader supply a confidence nobody has
Entirely below the threshold and also including zero	Evidence against an effect worth pursuing, since every plausible value falls short of the threshold and zero is among them. The cleanest outcome to report and the least often written up

Note what the third row does to the familiar reading of a non-significant result. “Not significant” does not establish that there is no effect; it can also mean the study could not have resolved an effect of the size you care about. Whether that is the situation is answerable by looking at the interval — if it extends well past the threshold, the question is still open — and better still by asking about power before running the test rather than arguing about it after. What it is not is automatically a sample-size problem: a narrow interval sitting entirely below the threshold is an informative negative, not an underpowered one.

Large samples cause the opposite trouble. Enough data makes almost any difference statistically detectable, so at scale statistical significance stops being informative and the threshold does all the work. The ASA’s principle applies directly: “Scientific conclusions and business or policy decisions should not be based only on whether a P-value passes a specific threshold.”

Why the range and not the number

The reason the table above is built on intervals is the one forecasting texts give: “If we only produce point forecasts, there is no way of telling how accurate the forecasts are.” A point estimate of 0.6% against a threshold of 0.5% and a point estimate of 0.6% with a range from −0.4% to 1.6% are the same number and different decisions.

Stating the effect size with its interval, next to the threshold, communicates in one line what a p-value and a verdict of statistically significant cannot — which is the practical form of the ASA’s warning that “by itself, a P-value does not provide a good measure of evidence regarding a model or hypothesis.”

The threshold shapes the test, not just the verdict

Naming the smallest effect worth detecting is also what makes it possible to size a test. A study designed for target power at a 5% change will generally have less power at a 1% change, increasing the risk of an inconclusive result. It can still detect a smaller effect; interpret the actual estimate and confidence interval.

So the threshold belongs at the start of experiment design, alongside the metric and the duration. Setting it first turns “how long should we run this” into arithmetic, and it occasionally produces the most useful answer available: the available traffic may provide too little expected precision to justify the cost. Consider redesigning, gathering more information, or deciding on other grounds; the calculation does not predetermine the result.

How this fits with sampling, uncertainty, bias, and the limits of experimental evidence is worked through in Reading a Result Honestly.

References:

• Wasserstein & Lazar, The ASA’s Statement on p-Values: Context, Process, and Purpose, The American Statistician 70:129–133, 2016 (principles as reproduced in Table I of this open-access discussion)
• Hyndman & Athanasopoulos, Forecasting: Principles and Practice (3rd ed.), §5.5 Prediction intervals