A Priori Power Analysis

A priori power analysis is a power calculation done before the data exists. You state the effect you want to be able to detect, the significance level you will use, and the power you want, and solve for the sample size. It is the version of the calculation that answers a question you can act on.

The NIST handbook’s answer to “how many measurements?” is the right starting point: “unfortunately, there is no correct answer without additional information (or assumptions).” Three things have to be supplied — α, β, and the population standard deviation.

The calculation

For a one-sample mean compared with a fixed reference, with independent normal observations and known population standard deviation \(\sigma\), the usual two-sided power approximation is

\[N = (z_{1-\alpha/2} + z_{1-\beta})^2 \left(\frac{\sigma}{\delta}\right)^2\]

Here \(N\) counts independent observations; for paired differences it counts independent pairs and \(\sigma\) is the standard deviation of the differences. For two independent groups, start from \(SE(\bar X_A-\bar X_B)=\sqrt{\sigma_A^2/n_A+\sigma_B^2/n_B}\). With equal variances and equal allocation, the corresponding normal approximation needs twice the one-sample \(N\) per group, four times that \(N\) in total. For example, \(\alpha=.05\), power .80, \(\sigma=1\), and \(\delta=.2\) give about 196.22 for one sample, 392.44 per group, or 784.89 total before rounding up. An actual design must account for variance estimation and allocation; a conversion-rate comparison also needs the baseline rate and an absolute or relative lift definition.

Three properties read straight off it. The quantity that matters is the ratio \(\sigma/\delta\) — the handbook notes that “it is usual to state the shift, \(\delta\), in units of the standard deviation,” which is an effect size. It is quadratic, so halving the detectable effect quadruples the sample. And both error rates enter additively inside the square, so a stricter significance level and a higher target power both cost observations.

A different goal has a different formula. If you only need an estimate precise to within \(\delta\) rather than a test, the requirement is \(N \ge (1.96/\delta)^2\sigma^2\) at 95% — a statement about a margin of error. Deciding which question you are asking precedes the arithmetic.

Three inputs, three ways to get them wrong

1. The effect size should be the smallest difference worth acting on, not the one you hope for. Sizing around an optimistic effect can leave low power for a smaller but important effect, increasing the risk of an inconclusive result.
2. The variability has to be assumed. The handbook flags the constraint — “a restriction is that the standard deviation must be known” — with the fallback being “the best estimate available from a previous experiment.” Since \(N\) scales with \(\sigma^2\), a 30% underestimate leaves you with roughly half the sample you needed. See sample standard deviation.
3. \(N\) must count independent units. Ten thousand events from two thousand users is not \(N = 10{,}000\) for a question about users — see sample size and IID.

When the sample is fixed

Often you cannot choose \(N\): you get the traffic you get, in the window you have. Then run the same equation backwards and ask what effect this design could detect — the minimum detectable lift. This states the effect corresponding to the chosen target power, not the smallest effect that could ever be detected. If the target-power lift is 12% and the plausible lift is 2%, expected power is lower; decide whether to redesign or proceed from likely precision, decision value, and cost.

Resist the shortcuts that look like they buy power. Loosening α buys it with false positives — “the risk \(\beta\) increases as the risk \(\alpha\) decreases,” and the reverse — and stopping early when a result looks good is a different procedure with a different error rate, not a faster version of this one. See risk of peeking.

Two habits make the output honest. Report it as a curve rather than a number — the detectable effect at two or three feasible sample sizes shows the trade to whoever decides how long to run, and does not imply a precision the assumed \(\sigma\) cannot support. And write the assumptions down with the answer, because the number is only as good as they are. Note also that the handbook describes the relationship between the true discrepancy and \(\beta\) as an “operating characteristic curve,” which is the honest object here too. For the after-the-fact version of the calculation and why it answers something else, see post hoc power analysis; for the form where neither error rate can be fixed, compromise power analysis.

References

• NIST/SEMATECH e-Handbook of Statistical Methods, §7.2.2.2 “Sample sizes required”
• §7.1.3 “What are statistical tests?”
• statsmodels, TTestIndPower.power: group sample sizes, allocation ratio, and standardized effect