Minimum Detectable Lift (MDL)

The minimum detectable lift is the effect size corresponding to a chosen target power for a specified design and significance level. It is not a hard detection boundary: smaller effects can be detected with lower probability, and even the target effect can be missed. Solving for the effect is useful when traffic or duration fixes the sample size.

For a one-sample mean against a fixed reference, with independent normal observations and known standard deviation \(\sigma\), rearranging the usual two-sided power approximation \(N = (z_{1-\alpha/2} + z_{1-\beta})^2(\sigma/\delta)^2\) gives

\[\delta = (z_{1-\alpha/2} + z_{1-\beta})\,\frac{\sigma}{\sqrt{N}}\]

Here \(N\) counts independent observations; for paired differences it counts independent pairs and \(\sigma\) is the standard deviation of the differences. For two independent groups, start from \(SE(\bar X_A-\bar X_B)=\sqrt{\sigma_A^2/n_A+\sigma_B^2/n_B}\). With equal variances and equal allocation, the corresponding normal approximation needs twice the one-sample \(N\) per group, four times that \(N\) in total. For example, \(\alpha=.05\), power .80, \(\sigma=1\), and \(\delta=.2\) give about 196.22 for one sample, 392.44 per group, or 784.89 total before rounding up. An actual design must account for variance estimation and allocation; a conversion-rate comparison also needs the baseline rate and an absolute or relative lift definition.

so the detectable effect shrinks only as \(1/\sqrt{N}\). At fixed significance level and target power, quadrupling the sample halves the minimum detectable effect — the single most useful fact for anyone deciding how long to run a test.

Why to compute it before running

Compute it to judge the expected information before paying for an experiment. A design with 80% power at a 12% effect has much lower power at a 2% effect, using the same effect units. That raises the risk of an inconclusive result; it does not guarantee one or mean the estimate contains no information. Assess likely precision, decision costs, and alternatives before choosing whether to run.

Comparing the detectable effect with the effect that would matter is the decision. Three outcomes, and each has a different response.

• Detectable effect well below what matters. The design has power to detect relevant effects under its assumptions. Run it if the expected information justifies the cost.
• Detectable effect above what matters. Consider more independent units, variance reduction, or another valid design. If those are unavailable, weigh the limited expected information against the cost rather than treating nondetection as certain.
• Close to the target effect. Report the planned detection probability and likely precision. Judge the eventual result by its estimate and interval, not by a presumed outcome.

What it depends on

The same three inputs as any sizing calculation, and the same hazards. \(\alpha\) and the target power set the multiplier, and they trade against each other — “the risk \(\beta\) increases as the risk \(\alpha\) decreases” — so a stricter threshold raises the detectable effect at fixed \(N\). \(\sigma\) must be assumed, and “a restriction is that the standard deviation must be known,” with a previous experiment as the usual source. And \(N\) must count independent units, not rows: see sample size and standard error.

One definitional care point. “Lift” is usually relative — a 2% lift on a 4% conversion rate means moving to 4.08%, not to 6% — so the absolute difference \(\delta\) and the relative lift are different numbers and the formula takes the absolute one. Stating which you mean prevents a factor-of-twenty misunderstanding. See conversion rate uplift.

Finally, keep this distinct from computing power after the fact from the observed effect. The detectable lift is a property of the design — sample size, variability, thresholds — and is legitimate to report at any time, including after a null result, precisely because it does not use the outcome. See post hoc power analysis for the version that does, a priori power analysis for the planning form, and trivial effects for the opposite failure, where a very large sample makes an irrelevant difference significant.

References

• NIST/SEMATECH e-Handbook of Statistical Methods, §7.2.2.2 “Sample sizes required”
• §7.1.3 “What are statistical tests?”

For two-group sizing, see statsmodels, TTestIndPower.power: group sample sizes, allocation ratio, and standardized effect.