Post Hoc Power Analysis

1. Definition

• Post hoc power analysis is a calculation of the statistical power of a test after the study has been completed, using:
  • The observed sample size (n)
  • The observed effect size (δ)
  • The chosen significance level (α)

It answers: “Given what we observed, what was the probability we could detect an effect?”

---

2. Why It’s Done

• Researchers sometimes use it when:
  • A study yields non-significant results → they want to know if this was due to low power.
  • Journals ask for a report on study sensitivity.
  • Meta-analysis needs to evaluate reliability of past studies.

---

3. Formula (Conceptual)

The calculation is the same as in a priori power analysis, but instead of assumed effect size, it uses the observed effect size:

$\text{Power} = P(\text{Reject H₀} \mid δ_{\text{observed}}, n, α)$

• If effect size observed = large → post hoc power will be high.
• If effect size observed = small → post hoc power will be low.

---

4. Example

Study Result

• H₀: Conversion rate = 10%
• Treatment group observed = 10.2% (tiny lift)
• Sample size = 1,000 per group
• α = 0.05

Post hoc power calculation might show only 12% power to detect such a tiny effect.

Interpretation: The study was underpowered — not enough users to reliably detect such a small improvement.

---

5. Limitations

• Tautology problem:
  • If results are non-significant, post hoc power will almost always be low.
  • If results are significant, post hoc power will almost always be high.
  • So it doesn’t add much beyond what p-value and effect size already tell you.
• Misinterpretation risk:
  • Some wrongly conclude “non-significant result + low power → null hypothesis is true.”
  • In reality, it only means “study wasn’t sensitive enough.”

---

6. Better Alternatives

• Instead of post hoc power, report:
  • Observed effect size (Cohen’s d, difference in proportions, odds ratio, etc.)
  • Confidence intervals for the effect.
• These give more meaningful information about the strength and precision of the result.

---

7. Comparison with A Priori Power Analysis

Feature	A Priori Power Analysis	Post Hoc Power Analysis
Timing	Before study	After study
Input	Assumed effect size	Observed effect size
Purpose	Determine required sample size	Explain why results may be non-significant
Value	Guides study design	Controversial; often redundant with p-value & CI

---

8. Key Takeaway

• A priori power analysis = essential for good planning.
• Post hoc power analysis = descriptive, but often redundant.
• Best practice: Report effect sizes + confidence intervals instead of relying on post hoc power.

---

In short:
Post hoc power analysis is done after a study to calculate achieved power using the observed effect size and sample size. While sometimes useful for describing sensitivity, it’s controversial because it often just restates the outcome of the significance test. Reporting effect size + confidence intervals is usually more informative.