Power Analysis

Power analysis is the calculation that connects four quantities — sample size, effect size, the significance level, and power — so that fixing any three determines the fourth. Most often it is used to answer “how many observations do I need?”, and the NIST handbook’s answer to that question is the right place to start:

Perhaps one of the most frequent questions asked of a statistician is, “How many measurements should be included in the sample?” Unfortunately, there is no correct answer without additional information (or assumptions).

There is no sample size without assumptions. Anyone who gives you a number without asking what you are trying to detect has substituted a convention for a calculation.

The three inputs

The handbook lists what the sample size for a test about a mean depends on: “value selected for \(\alpha\), the risk of rejecting a true hypothesis,” “value of \(\beta\), the risk of accepting a false null hypothesis when a particular value of the alternative hypothesis is true,” and “value of the population standard deviation.”

For a one-sample mean compared with a fixed reference, with independent normal observations and known population standard deviation \(\sigma\), the usual two-sided power approximation is

\[N = (z_{1-\alpha/2} + z_{1-\beta})^2 \left(\frac{\sigma}{\delta}\right)^2\]

Here \(N\) counts independent observations; for paired differences it counts independent pairs and \(\sigma\) is the standard deviation of the differences. For two independent groups, start from \(SE(\bar X_A-\bar X_B)=\sqrt{\sigma_A^2/n_A+\sigma_B^2/n_B}\). With equal variances and equal allocation, the corresponding normal approximation needs twice the one-sample \(N\) per group, four times that \(N\) in total. For example, \(\alpha=.05\), power .80, \(\sigma=1\), and \(\delta=.2\) give about 196.22 for one sample, 392.44 per group, or 784.89 total before rounding up. An actual design must account for variance estimation and allocation; a conversion-rate comparison also needs the baseline rate and an absolute or relative lift definition.

where \(\delta\) is “the difference or shift we want to detect.” Three features of that formula are worth reading off it directly.

• The ratio \(\sigma/\delta\) is what matters, not \(\delta\) alone — the handbook notes that “it is usual to state the shift, \(\delta\), in units of the standard deviation,” which is exactly an effect size.
• It is quadratic. Halving the effect you want to detect multiplies the required sample by four. This is why chasing small effects gets expensive so fast.
• Both error rates appear, additively, inside the square. A stricter \(\alpha\) and a higher target power both raise \(N\) — see significance level and statistical power.

A different question has a simpler answer. If you only want an estimate precise to within \(\delta\) rather than a test, the requirement is \(N \ge (1.96/\delta)^2\sigma^2\) at 95% — a statement about a margin of error rather than about detecting a hypothesis. Deciding which of the two questions you are asking comes before any arithmetic.

Three directions, one equation

You fix	You solve for	When this is the right form
Effect size, \(\alpha\), power	Sample size	Planning a study you can size freely — see a priori power analysis
Sample size, \(\alpha\), power	Detectable effect	The sample is what it is — traffic, patients, a fixed window. This is the most useful form in practice
Sample size, effect size, ratio of \(\alpha\) to \(\beta\)	Both error rates	Neither can be fixed independently — see compromise power analysis

The second row deserves emphasis because teams reach for the first when the second describes their situation. If you get 40,000 visitors a week and the decision is due in two weeks, sample size is not a free variable; the question is what that sample can detect, and whether that is small enough to be worth running. A design targeting 80% power at a 12% lift may have much lower power at a plausible 2% lift; detection is still possible, and the estimate may still be useful. Compare expected precision and decision value with the cost before committing the two weeks. Use the two-group calculation and define whether lift is absolute or relative. See minimum detectable lift.

The input people skip

Variability is required and unknown. The handbook is blunt about the constraint — “a restriction is that the standard deviation must be known” — and about the workaround: “lacking an exact value for the standard deviation requires some accommodation, perhaps the best estimate available from a previous experiment.”

In practice that means one of three things, and all three should be stated rather than buried: an estimate from historical data on the same metric, a pilot, or a deliberately conservative upper bound. Since \(N\) scales with \(\sigma^2\), a 30% underestimate of the standard deviation leaves you with roughly half the sample you needed — and the study still runs, still reports, and has lower power to detect the specified effect; failure to detect it is more likely, but not certain. See sample standard deviation.

Two further cautions on the unit of analysis, both of which invalidate the arithmetic rather than bias it slightly. \(N\) must count independent units: ten thousand events from two thousand users is not \(N = 10{,}000\) for a question about users, which is the same trap as in standard error. And the effect size you assume should be the smallest difference worth acting on, which is a business judgment — using an effect you hope for produces a study sized for optimism.

Finally, treat the output as a curve rather than a number. A single required \(N\) invites the impression of precision that the assumed \(\sigma\) does not support; reporting the detectable effect at two or three feasible sample sizes makes the trade visible to whoever decides how long to run. See sample size for the sizing decision itself and A/B testing for the setting where it is used most.

References

• NIST/SEMATECH e-Handbook of Statistical Methods, §7.1.3 “What are statistical tests?”
• §7.2.2.2 “Sample sizes required”

For two-group sizing, see statsmodels, TTestIndPower.power: group sample sizes, allocation ratio, and standardized effect.