1. Definition
- Effect size (δ, delta) measures the magnitude of a difference or relationship, not just whether it is statistically significant.
- While a p-value tells you if an effect exists, effect size tells you how big the effect is.
- It is essential for understanding practical importance and for power analysis (sample size planning).
2. Why It Matters
- Statistical significance ≠ practical significance.
- Example: With a huge sample, a 0.1% improvement may be “significant” but meaningless in practice.
- Effect size helps quantify real-world impact.
- Used in:
- Experimental design (sample size calculation).
- Reporting research results (e.g., medicine, psychology).
- A/B testing in business.
3. Types of Effect Sizes
Different definitions depending on the statistical test:
(a) Cohen’s d (Standardized Mean Difference)
For comparing two means:
$d = \frac{\bar{X}_1 – \bar{X}_2}{s_p}$
where $s_p$ = pooled standard deviation.
Interpretation (Cohen’s rule of thumb):
- 0.2 = small, 0.5 = medium, 0.8 = large.
(b) δ (Noncentrality Parameter in t-tests)
In power analysis, δ often denotes the standardized effect size:
$δ = \frac{μ – μ_0}{σ / \sqrt{n}}$
- μ = true mean
- μ₀ = hypothesized mean under H₀
- σ = population standard deviation
- n = sample size
This δ is basically the expected t-statistic under H₁.
- Larger δ → stronger ability to detect differences (higher power).
(c) Correlation Effect Size (r, R², η²)
- r: Pearson correlation (strength of relationship).
- R²: Proportion of variance explained in regression.
- η² (eta-squared): Effect size for ANOVA.
(d) Proportion Effect Size
For proportions (A/B testing):
$δ = p_1 – p_2$
or standardized version:
$h = 2 \arcsin(\sqrt{p_1}) – 2 \arcsin(\sqrt{p_2})$
4. Example
One-Sample t-test
- Population mean under H₀ = 100.
- Sample mean = 104, σ = 10, n = 25.
Effect size δ:
$δ = \frac{104 – 100}{10 / \sqrt{25}} = \frac{4}{2} = 2.0$
δ = 2.0 means the observed mean is 2 standard errors away from H₀.
- Very large effect → likely to be statistically significant.
5. Effect Size vs P-value
| Aspect | P-value | Effect Size (δ) |
|---|---|---|
| Tells you | Whether an effect exists (statistical significance) | How big the effect is (practical significance) |
| Depends on | Sample size (large n → small effects become significant) | Independent of sample size (focuses on magnitude) |
| Use | Hypothesis testing | Interpretation, power analysis, meta-analysis |
6. Interpretation in Practice
- Small δ (≈0.2): Weak but possibly meaningful in large populations (e.g., public health).
- Medium δ (≈0.5): Moderate difference, often noticeable.
- Large δ (≥0.8): Strong, highly impactful difference.
In short:
- Effect size (δ) quantifies the strength of a difference or relationship.
- It complements the p-value by answering “How big is the effect?”
- In power analysis, δ is the standardized mean difference, crucial for calculating required sample sizes.