Effect Size (δ)

1. Definition

• Effect size (δ, delta) measures the magnitude of a difference or relationship, not just whether it is statistically significant.
• While a p-value tells you if an effect exists, effect size tells you how big the effect is.
• It is essential for understanding practical importance and for power analysis (sample size planning).

---

2. Why It Matters

• Statistical significance ≠ practical significance.
  • Example: With a huge sample, a 0.1% improvement may be “significant” but meaningless in practice.
• Effect size helps quantify real-world impact.
• Used in:
  • Experimental design (sample size calculation).
  • Reporting research results (e.g., medicine, psychology).
  • A/B testing in business.

---

3. Types of Effect Sizes

Different definitions depending on the statistical test:

(a) Cohen’s d (Standardized Mean Difference)

For comparing two means:

$d = \frac{\bar{X}_1 – \bar{X}_2}{s_p}$

where $s_p$​ = pooled standard deviation.

Interpretation (Cohen’s rule of thumb):

• 0.2 = small, 0.5 = medium, 0.8 = large.

---

(b) δ (Noncentrality Parameter in t-tests)

In power analysis, δ often denotes the standardized effect size:

$δ = \frac{μ – μ_0}{σ / \sqrt{n}}$

• μ = true mean
• μ₀ = hypothesized mean under H₀
• σ = population standard deviation
• n = sample size

This δ is basically the expected t-statistic under H₁.

• Larger δ → stronger ability to detect differences (higher power).

---

(c) Correlation Effect Size (r, R², η²)

• r: Pearson correlation (strength of relationship).
• R²: Proportion of variance explained in regression.
• η² (eta-squared): Effect size for ANOVA.

---

(d) Proportion Effect Size

For proportions (A/B testing):

$δ = p_1 – p_2$

or standardized version:

$h = 2 \arcsin(\sqrt{p_1}) – 2 \arcsin(\sqrt{p_2})$

---

4. Example

One-Sample t-test

• Population mean under H₀ = 100.
• Sample mean = 104, σ = 10, n = 25.

Effect size δ:

$δ = \frac{104 – 100}{10 / \sqrt{25}} = \frac{4}{2} = 2.0$

δ = 2.0 means the observed mean is 2 standard errors away from H₀.

• Very large effect → likely to be statistically significant.

---

5. Effect Size vs P-value

Aspect	P-value	Effect Size (δ)
Tells you	Whether an effect exists (statistical significance)	How big the effect is (practical significance)
Depends on	Sample size (large n → small effects become significant)	Independent of sample size (focuses on magnitude)
Use	Hypothesis testing	Interpretation, power analysis, meta-analysis

---

6. Interpretation in Practice

• Small δ (≈0.2): Weak but possibly meaningful in large populations (e.g., public health).
• Medium δ (≈0.5): Moderate difference, often noticeable.
• Large δ (≥0.8): Strong, highly impactful difference.

---

In short:

• Effect size (δ) quantifies the strength of a difference or relationship.
• It complements the p-value by answering “How big is the effect?”
• In power analysis, δ is the standardized mean difference, crucial for calculating required sample sizes.