Effect Size (δ)

An effect size measures how large a difference or relationship is, in standardized or original units. It answers a question a significance test does not: not “would data like this be surprising under the null model?” but “how big is the thing we found?”

The American Statistical Association put the reason for keeping them separate in one sentence, as the fifth of its six principles on p-values: “A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.” Two distinct quantities, two distinct jobs.

The families of effect size

Which one you use depends on what you are comparing. The most common for two means is the standardized mean difference, Cohen’s \(d\):

\[d = \frac{\bar{X}_1 - \bar{X}_2}{s_p}\]

where \(s_p\) is the pooled standard deviation. The denominator is a standard deviation. That is the definition, and the next section is about what happens when a standard error is used there instead.

• Relationships: the correlation \(r\) for strength of association (see correlation), \(R^2\) for the share of variance a regression accounts for, \(\eta^2\) for ANOVA.
• Proportions: the raw difference \(p_1 - p_2\), which in an A/B test is usually the quantity a decision is actually about, or a standardized version such as Cohen’s \(h = 2\arcsin(\sqrt{p_1}) - 2\arcsin(\sqrt{p_2})\) when you need one that behaves consistently across the range of \(p\).

Cohen’s conventions — about 0.2 small, 0.5 medium, 0.8 large — are widely quoted for \(d\). They are conventions proposed for behavioral research when nothing better is available, not properties of the world. In a business context a \(d\) of 0.1 can be worth a great deal and a \(d\) of 1.0 can be irrelevant, so the useful comparison is against a threshold your domain sets rather than against the labels.

The denominator decides which quantity you get

Here is the mistake this entry exists to prevent. Take a one-sample comparison: the null mean is 100, the sample mean is 104, the standard deviation is 10, and \(n = 25\). Three different numbers come out of those figures.

Quantity	Computation	Value	Depends on \(n\)?
Raw difference	\(104 - 100\)	4 units	No
Effect size (standardized)	divide by the standard deviation: \(4/10\)	0.4	No
Test statistic	divide by the standard error: \(4/(10/\sqrt{25})\)	2.0	Yes

Both divisions look alike on paper and differ by a factor of \(\sqrt{n}\). Read the 2.0 as an effect size and you have a “very large effect” by any rule of thumb; read it correctly and you have an estimated standardized effect of 0.4, which sits between small and medium, observed at two standard errors from the null.

The internal contradiction is the tell. For a fixed population difference and standard deviation, the standardized population effect does not change merely because \(n\) changes, while the test statistic scales with \(\sqrt{n}\). Sample effect estimates can fluctuate or have finite-sample bias as more data are collected. Comparisons across studies also need compatible outcomes, populations, and standardization choices.

One detail on the 10: if it is a known population \(\sigma\) the statistic is a \(z\); if it is a sample standard deviation the same arithmetic gives a \(t\) with \(n - 1 = 24\) degrees of freedom, and the standardized effect is an estimate \(\hat{d} = 0.4\). The distinction changes the reference distribution, not the arithmetic here.

The δ that is not an effect size

The symbol is overloaded, and that is the second way this goes wrong. In power analysis δ commonly denotes the noncentrality parameter, which is an effect size multiplied by the square root of the sample size:

\[\delta = \frac{\mu - \mu_0}{\sigma}\sqrt{n} = d\sqrt{n}\]

It already contains \(n\) by construction, which is exactly how a larger sample produces higher power for the same underlying effect — and exactly why it is not an effect size.

It is also a planning input computed from assumed population values before any data exists, not something you calculate from a result. Assume a true effect of \(0.5\sigma\) and plan \(n = 25\), and the noncentrality is \(0.5 \times 5 = 2.5\); power is then the probability that the statistic exceeds the critical value under the noncentral distribution that 2.5 locates. That 2.5 is a decision about what to look for. The 2.0 in the table above is a measurement. Substituting an observed mean and a sample standard deviation into the formula gives an estimate of the parameter, and the parameter itself is fixed and unknown while the statistic varies from sample to sample.

A related approximation is worth stating precisely, because “δ is basically the expected t statistic” is repeated often enough to mislead. It is close to the mean of the noncentral distribution when the degrees of freedom are large, and not equal to it: at \(\nu = 24\) and \(\delta = 2.5\) the mean is about 2.58, because the exact mean carries a gamma-function factor.

The practical rule: before putting a number into a power formula, check which δ it wants. Feeding a standardized mean difference to a formula expecting the noncentrality parameter, or the reverse, changes the answer by a factor of \(\sqrt{n}\) — and nothing in the output looks wrong. See power analysis and statistical power.

Effect size and p-value

Aspect	P-value	Effect size
Tells you	How incompatible the data are with a specified statistical model — not whether an effect exists, and not the probability that it does	How big the observed effect is, in standardized or original units
Sample size	Depends on it: with large \(n\), small effects reach any threshold	Population definition does not scale with n; sample estimates can vary with sampling and bias
Used for	A decision rule about error rates	Interpretation, sizing an experiment, meta-analysis

The first row follows the ASA’s own wording. Its first principle is that “p-values can indicate how incompatible the data are with a specified statistical model,” and its second rules out the two readings people reach for: p-values “do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.” See p-value and statistically significant.

Which is why the two belong together in a report rather than in sequence. A significant result with an effect size too small to act on and a non-significant result consistent with an effect large enough to matter are both common, and both are invisible if only one of the numbers is shown. The ASA’s third principle makes the general form of the point: decisions “should not be based only on whether a p-value passes a specific threshold.”

Where it does real work

Beyond reporting, the effect size is the input that makes experiment planning possible. You cannot ask how many observations you need without first saying how small an effect you want to be able to detect, which is a business judgment before it is a statistical one — see sample size and A/B testing.

It is also what makes results comparable. Two studies measuring the same thing in different units, or the same metric at different scales, can be put side by side in standardized units — the basis of meta-analysis, and the reason effect sizes rather than p-values are what accumulate into knowledge.

References

• American Statistical Association, “ASA Releases Statement on Statistical Significance and P-Values” (2016).
• Jacob Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed. (Lawrence Erlbaum, 1988).