Statistical Power

1. Core Definition

Statistical power is

the probability of correctly detecting a real effect when it truly exists.

• Formula: Power = 1 − β
• β (Type II error):
  • The probability of failing to detect an effect that actually exists

Power measures detection ability, not correctness of conclusions.

2. The Full Hypothesis Testing Framework

Power answers the question:
“If an effect is real, how likely am I to detect it?”

3. Intuition: Signal vs Noise

Key idea

Power is fundamentally a signal-to-noise problem.

• Signal: effect size
• Noise: standard error (variability / √n)

Roughly,$$\text{Test statistic} \approx \frac{\text{Effect}}{\text{Standard Error}}$$

• Larger values → higher probability of exceeding the critical value → higher power

4. Power from a Distribution Perspective

Hypothesis testing always compares two distributions:

(1) Distribution under H0

• Effect = 0
• Test statistic centered near zero

(2) Distribution under H1

• Effect ≠ 0
• Distribution shifts away from zero

Power
= the area of the H1 distribution beyond the critical value.

5. Four Key Determinants of Power

① Effect Size

• Larger effect → higher power
• Common measures:
  • Mean difference (Cohen’s d)
  • Correlation (r)
  • Explained variance (R²)

② Sample Size (n)

• Larger n → smaller standard error → higher power
• Standard error ∝ 1 / √n

③ Variability / Noise (σ)

• More variability → lower power
• Cleaner, more precise data improves detection

④ Significance Level (α)

• Higher α → higher power
• Trade-off: increased Type I error risk

6. Common Misconceptions

Misconception 1

“Power = 80% means the result is 80% correct.”

Reality:

• Power does not measure correctness
• It measures the ability to detect a real effect

Misconception 2

“A non-significant result means no effect.”

Reality:

• Non-significant means insufficient evidence
• The study may simply lack power

Misconception 3

“Bigger samples are always better.”

Reality:

• Large samples can make trivial effects statistically significant
• Statistical significance ≠ practical significance

7. Why Power Matters

1) Better understanding of p-values

• p-values describe outcomes
• power describes capability

2) Understanding study design

• Explains why sample size, measurement precision, and modeling choices matter

3) Reframing “failure”

• Failure to reject H0 does not imply a flawed study
• It may indicate insufficient detection power

8. Ways to Increase Power

9. Conventional Benchmark

• Common target: Power = 0.80
• Meaning:
  • If an effect truly exists, there is an 80% chance of detecting it

10. Key Takeaway Statements

• Power measures analytical capability
• Non-significant ≠ no effect
• Power matters both before and after analysis
• Statistics is about managing uncertainty, not guaranteeing truth

11. Concept Map

Effect size ↑
Sample size ↑    →  Signal / Noise ↑  → Power ↑
Variance ↓
α ↑

Effect size ↑
Sample size ↑    →  Signal / Noise ↑  → Power ↑
Variance ↓
α ↑