1. Definition
- Minimum Detectable Lift (MDL) = the smallest relative improvement (percentage increase or decrease) in a key metric (conversion rate, revenue, clicks, etc.) that your experiment can reliably detect, given your:
- Sample size (n)
- Significance level (α)
- Power (1 – β)
It’s essentially the smallest effect size you care about detecting.
2. Why It Matters
- Prevents over-optimizing for trivial effects.
- Ensures experiments are designed with enough sample size to detect meaningful changes.
- Helps product teams decide: “What improvement do we need to justify running this test?”
3. Formula (Conceptual)
For conversion rates:
$\text{MDL} = \frac{p_{\text{treatment}} – p_{\text{control}}}{p_{\text{control}}}$
where:
- $p_{\text{control}}$ = baseline conversion rate.
- $p_{\text{treatment}}$ = conversion rate in variant.
The smaller the MDL, the larger the sample size required.
4. Example
Website A/B Test
- Baseline conversion (control) = 5% (0.05)
- Suppose test is designed with α = 0.05, power = 0.80, n = 20,000 visitors per variant.
- Using sample size formulas, you calculate:
- The experiment can detect a 10% relative lift (from 5% → 5.5%).
This means the Minimum Detectable Lift (MDL) = +10%.
- If the true lift is smaller (say +2%), the test likely won’t detect it with significance.
5. MDL vs MDE
- Sometimes you’ll see Minimum Detectable Effect (MDE).
- MDE = absolute effect size (e.g., +0.5 percentage points).
- MDL = relative effect size (percentage lift).
Example:
- Control = 5%
- Treatment = 5.5%
- MDE = +0.5 percentage points
- MDL = +10% relative lift
6. Interpretation in Practice
- If MDL is too high → you might miss smaller but still valuable improvements.
- If MDL is too low → experiment may require millions of users → impractical.
- Balance is needed:
- Business sets minimum improvement worth acting on.
- Statistician designs experiment to detect at least that lift.
7. Summary Table
| Term | Meaning | Example |
|---|---|---|
| MDE (Minimum Detectable Effect) | Smallest absolute effect detectable | +0.5% conversion points |
| MDL (Minimum Detectable Lift) | Smallest relative effect detectable | +10% lift over 5% baseline |
In short:
Minimum Detectable Lift (MDL) is the smallest relative improvement an experiment can reliably detect given α, power, and sample size. It ensures you design tests that are big enough to detect meaningful business improvements, but not so sensitive that you waste time detecting trivial ones.