What is Power Analysis?

Power analysis is the process of determining how large a sample size you need to reliably detect an effect in an experiment.

It balances 4 key quantities in hypothesis testing:

  1. Effect size (δ) → The minimum difference you care about (e.g., 2% uplift in conversion).
  2. Significance level (α) → The probability of a false positive (commonly 0.05).
  3. Power (1 – β) → The probability of correctly detecting a true effect (commonly 0.8 or 80%).
  4. Sample size (n) → Number of observations needed.

If you know any 3, you can calculate the 4th.

Why It Matters

  • Too small sample → test underpowered → you might miss a real effect.
  • Too large sample → waste of time/money, may detect trivial effects that aren’t practically useful.
  • Power analysis ensures experiments are efficient and interpretable.

Example

Suppose you run an A/B test on conversion rates:

  • Baseline conversion rate (control) = 10%.
  • Minimum detectable lift (effect size) = +2% (absolute) → want to detect 12% vs 10%.
  • Significance level (α) = 0.05.
  • Desired power = 0.8.

Using a two-proportion z-test power analysis → you need about 3,200 users per group.

So total sample size = 6,400 users for the experiment.

Formula (simplified for two-proportion test)

For comparing two proportions:

$n = \frac{(Z_{1-\alpha/2}\sqrt{2p(1-p)} + Z_{1-\beta}\sqrt{p_1(1-p_1) + p_2(1-p_2)})^2}{(p_1 – p_2)^2}$

Where:

In Python (statsmodels)

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# Parameters
p1, p2 = 0.10, 0.12   # 10% vs 12%
alpha = 0.05
power = 0.8

effect_size = proportion_effectsize(p1, p2)
analysis = NormalIndPower()
sample_size = analysis.solve_power(effect_size, power=power, alpha=alpha, ratio=1)

print("Required sample size per group:", sample_size)

Types of Power Analysis

  1. A priori → Calculate needed sample size before experiment (most common).
  2. Post hoc → Calculate achieved power after experiment.
  3. Compromise → Balance between α and β when resources are fixed.

Summary

  • Power analysis ensures your experiment is big enough to detect a meaningful effect, but not wastefully large.
  • Inputs: effect size, α, desired power.
  • Output: required sample size.