Statistical Power

Statistical power is the probability that a test rejects the null hypothesis when a specified alternative is true — written \(1-\beta\), where \(\beta\) is the chance of missing it. It answers the question a p-value cannot: if the effect were there, would this study have found it?

The phrase to hold on to is “a specified alternative.” Power is not a property of a test; it is a property of a test together with an effect size you have named. “This study is underpowered” is incomplete until you say underpowered to detect what.

Why β is not yours to set

The NIST handbook draws the asymmetry sharply. You choose \(\alpha\); \(\beta\) “is not chosen by the user but is determined, as one might expect, by the magnitude of the real discrepancy.” And it behaves as you would hope: “large discrepancies between reality and the null hypothesis are easier to detect and lead to small errors of the second kind; while small discrepancies are more difficult to detect and lead to large errors of the second kind.”

Which is why power has to be quoted against a target. The same experiment can have 95% power to detect a 10% lift and 20% power to detect a 1% lift, and both numbers are correct. The smallest effect a design can reliably find is its minimum detectable lift, and the input you name is an effect size.

Four things move power, and it is worth knowing which of them you control.

The last row is the trade the handbook states outright: “the risk \(\beta\) increases as the risk \(\alpha\) decreases.” You can buy power by loosening the threshold, which is the same as buying it with false positives. At fixed \(\alpha\), increasing \(n\) can lower \(\beta\), but does not automatically lower the Type I error rate. More information, from additional independent observations or lower variance, can instead support a redesigned test with a smaller \(\alpha\) at the same target power — see Type I error for the other side of the ledger.

The reason this matters: reading a null result

Power is what makes a non-significant result interpretable, and the handbook names both possibilities in one sentence: “not rejecting may be a good result if we want to continue to act as if we ‘believe’ the null hypothesis is true. Or it may be a disappointing result, possibly indicating we may not yet have enough data to ‘prove’ something.”

Nothing in the test output separates the two. High planned power describes the chance of rejecting under the specified alternative; it does not turn a nonsignificant result into a posterior probability that the effect is small. Use the effect estimate and its confidence interval to assess which magnitudes remain compatible with the data. A low-powered study may leave a wide range unresolved.

The practical way to read it is as an interval rather than a verdict. A confidence interval that spans a 5% loss to a 6% gain says plainly that the study could not distinguish the outcomes you care about; the word “not significant” hides exactly that. See statistically significant and hypothesis testing.

Before the data, not after

Power calculations belong to design. Computing power afterward from the observed effect — sometimes called observed or post hoc power — is close to circular: a result that just missed significance will always come out as low power, because both quantities are functions of the same statistic. It cannot tell you whether a real effect was missed. See post hoc power analysis for what the term does and does not support, and a priori power analysis for the version that answers a real question.

A note on the familiar 80%. It is a convention, and the handbook offers no such number — it describes the whole relationship between the true discrepancy and \(\beta\) as an “operating characteristic curve,” which is the honest object: a curve, not a target. Whether 80% is enough depends on what a miss costs relative to a false positive, which returns the question to the same domain judgment as \(\alpha\).

How the target power turns into a required number of observations is power analysis. For online experiments, where the sample accumulates over time and the temptation is to look early, the design question also includes when you are allowed to stop — see risk of peeking and A/B testing.

Repeated observations and effective sample size

Repeated sessions from one user need not supply independent information. For N independent users, each with m observations of common variance σ² and exchangeable within-user correlation ρ, the variance of the grand mean is σ²[1+(m−1)ρ]/(Nm). Matching that to σ²/n_eff gives n_eff = Nm/[1+(m−1)ρ]. This derivation assumes equal cluster sizes and this particular mean estimator.

With 12 sessions per user, ρ = 0.1 gives n_eff ≈ 5.71N, while ρ = 0.6 gives about 1.58N, rather than 12N independent sessions. Session-level outcomes remain usable with user-level assignment, but uncertainty must account for the user clusters. Unequal cluster sizes, weighting and ratio metrics need an estimator-specific variance calculation; effective sample size is not simply the user count. See Bakshy and Eckles on dependence in online experiments for the inference problem; the equal-cluster formula here is the variance derivation stated above.

Further reading

• Bakshy and Eckles (2013), Uncertainty in Online Experiments with Dependent Data
• NIST/SEMATECH e-Handbook of Statistical Methods, §7.1.3 “What are statistical tests?”
• §7.2.2.2 “Sample sizes required”