Power (1 – β)

Power is the probability that a test rejects the null hypothesis when a specified alternative is true. It equals \(1 – \beta\), where \(\beta\) is the probability of a Type II error, failing to reject a null hypothesis that is false. statsmodels’ documentation for TTestIndPower.power puts it the same way: the power of a test, for example 0.8, is one minus the probability of a type II error.

Power belongs to a test together with a named effect size. Saying a study has 80% power means 80% power to detect that effect with that design.

What sets it

NIST/SEMATECH e-Handbook §7.1.3 notes that \(\beta\) is not chosen by the user. It is determined by the magnitude of the real discrepancy, and the risk \(\beta\) increases as the risk \(\alpha\) decreases. Four things move power:

• Effect size. Larger discrepancies are easier to detect.
• Sample size. More data narrows the sampling variation.
• Significance level. A smaller \(\alpha\) makes rejection harder, so power falls.
• Variability. A larger standard deviation hides a given difference.

For a test of a mean with a known standard deviation and a normal approximation, NIST/SEMATECH e-Handbook §7.2.2.2 gives the two-sided sample size \(N = (z_{1-\alpha/2} + z_{1-\beta})^2 (\sigma/\delta)^2\), where \(\delta\) is the difference to detect. When \(\sigma\) is unknown the calculation uses the t distribution and has to be iterated.

A computed example

The design is hypothetical. A two-sample t-test with equal group sizes and equal variances, a two-sided \(\alpha = 0.05\), and a standardized effect size of 0.5, meaning the difference between the means is half a standard deviation.

Reaching 80% power takes about 64 per group. The normal approximation gives 63, and the t-test needs one more. If the true effect is 0.2 instead of 0.5, about 394 per group are needed for 80% power.

The design matters as much as the numbers. A one-sample t-test with the same standardized effect of 0.5 and 30 observations has power of about 0.75, much higher than the two-sample case at 30 per group, because its standard error is smaller: it estimates one mean, not a difference between two.

Using it

• Decide the effect worth detecting and the target power before collecting data. Power Analysis and A Priori Power Analysis cover the calculation.
• A result that is not significant from a low-power study is ambiguous: it may mean no effect or a missed one. See Statistical Significance.
• For what can be done after the data are in, see Post Hoc Power Analysis.
• The complementary risk of a false positive is Type I Error. The longer treatment of this idea is Statistical Power.

References:

• NIST/SEMATECH e-Handbook of Statistical Methods, §7.1.3, What are statistical tests?
• NIST/SEMATECH e-Handbook of Statistical Methods, §7.2.2.2, Sample sizes required
• statsmodels documentation, TTestIndPower.power