Conversion Rate Uplift

Conversion rate uplift is the change in a conversion rate caused by a variant, measured against the control it replaced. Kohavi and colleagues give the underlying metric its plain definition — the conversion rate of an e-commerce site is “the percentage of visits to the website that include a purchase” — and the unit in that sentence is part of the definition. A rate per visit and a rate per user are different numbers on the same data, and an uplift computed on one does not transfer to the other.

Two numbers, one result

With \(CR_A\) the control rate and \(CR_B\) the treatment rate, the absolute uplift is \(CR_B - CR_A\), measured in percentage points, and the relative uplift is \((CR_B - CR_A)/CR_A\), measured as a percent of the baseline. They describe the same result and differ by a factor of \(1/CR_A\), which on a 4 percent baseline is a factor of twenty-five.

Control	Treatment	Absolute uplift	Relative uplift
5.0%	6.0%	+1.0 percentage point	+20%
10.0%	9.5%	-0.5 percentage points	-5%
2.0%	3.0%	+1.0 percentage point	+50%
4.0%	4.08%	+0.08 percentage points	+2%

The first and third rows move the rate by the same amount and are reported as +20 percent and +50 percent. The last row is the one that causes arguments: a request for a 2 percent lift on a 4 percent baseline usually means 4.08 percent, and is sometimes read as 6 percent. Because the ambiguity lives in the word rather than in the arithmetic, the fix is to name the scale every time — one percentage point, or twenty percent — and never to write a bare percent sign for the difference of two rates.

Relative uplift is the form usually reported, for the reason Kohavi and colleagues give: “For many online metrics, the difference in the means is so small that percent change has much more intuitive meaning than the absolute difference.” Their example is a clickthrough rate whose treatment effect was 0.00014, which became a 12.85 percent change, and they note that “the latter number was much more meaningful to decision makers”. Small absolute effects are the norm rather than the exception; Radcliffe and Surry describe a control rate around 1 percent against a treated rate around 1.1 percent as typical of marketing response work.

Which one the arithmetic needs

Design calculations take the absolute difference. Sample-size and power formulas are written in terms of a difference \(\delta\) measured in the same units as the outcome, so a target stated as a relative lift has to be converted before it can be used: a 2 percent lift on a 4 percent baseline enters the formula as 0.0008, not as 0.02. The minimum detectable lift entry works through the conversion and the design questions around it; sample size and statistical power cover what the formula is doing.

Reporting goes the other way, and the two conventions meeting in one document is where a factor of the baseline goes missing. A launch memo that says the test targeted a 2 percent lift and delivered 0.9 points has quietly changed units between two sentences.

Whether the difference is real

An observed rate difference is an estimate. Random assignment supports causal interpretation when observation and analysis preserve the design; observational comparisons need defensible identification assumptions and adjustment — see A/B testing. For independent binary observations with adequate counts, one option is the two-proportion z-test, and the interval for the absolute difference is the usual one, \(CR_B - CR_A \pm 1.96\,\hat{\sigma}_d\). For a confidence interval obtained by inverting the same two-sided test, inclusion of zero corresponds to non-rejection at the matching significance level. A pooled z-test and an unpooled Wald interval use different variance estimates and need not agree exactly.

Not rejecting is a statement about the evidence, not a finding that the rates are equal. A test with wide intervals is compatible with a large effect in either direction, and reporting it as no difference converts missing evidence into a conclusion. The statistically significant and p-value entries cover what the result does and does not license.

An interval for the relative figure

Here the two scales stop being interchangeable. Dividing the endpoints of the absolute interval by the observed control rate looks like a conversion of units and is not one. Kohavi and colleagues state the reason: “forming a confidence interval around the percent change is not a straightforward extension of the confidence interval for the absolute effect. This is because we are now dividing by a random variable.” The control rate in the denominator is itself an estimate, and its uncertainty belongs in the interval. The derivation they cite is Fieller’s, and the resulting interval is asymmetric about the point estimate.

The numbers below are illustrative. Two arms of 50,000 visits each, 2,500 conversions in the control and 2,650 in the treatment, giving 5.000 percent against 5.300 percent — an absolute uplift of 0.300 percentage points and a relative uplift of 6.00 percent. The pooled two-proportion z-test gives \(z = 2.1462\) and \(p = 0.0319\).

Quantity	95% interval
Absolute uplift	+0.026 to +0.574 percentage points
Relative uplift, absolute interval divided by the control rate	+0.52% to +11.48%
Relative uplift, Fieller	+0.51% to +11.80%
Relative uplift, delta method	+0.36% to +11.64%

At this sample size the displayed endpoints are fairly close, but that numerical agreement does not establish the shortcut’s coverage. It stops being harmless as the arms shrink, because what governs the difference is the coefficient of variation of the control rate — its standard error divided by its value. The same 5 percent baseline with 500 visits per arm and a 6.4 percent treatment rate gives +28.0 percent relative uplift with a Fieller interval of -23.8 to +123.6 percent, a naive interval of -29.5 to +85.5 percent, and a delta-method interval of -37.1 to +93.1 percent. Fieller’s upper bound is about 45 percent above the naive one, and the delta-method interval is symmetric about the point estimate, which the sampling distribution of a ratio is not.

Push further and the finite interval stops existing. What fails is this construction, not the method: a general Fieller confidence set can be unbounded, or the complement of a bounded interval, and still have the right coverage. The function below returns only the finite case. Kohavi and colleagues put the boundary in operational terms: “if the denominator is stochastically close to zero one or both endpoints will not exist. In practice, you shouldn’t calculate this interval if the confidence interval for the denominator contains zero.” Fieller’s denominator carries the factor \(1 - 1.96^2 \cdot CV_A^2\), which reaches zero when the control rate’s coefficient of variation reaches \(1/1.96\). For a proportion that is \(N = 1.96^2 (1 - p)/p\) observations, which at a 5 percent baseline is 73 control visits. Below that boundary, the finite-interval formula used here no longer applies. A general confidence-set method can still express the uncertainty; a first-order delta approximation can also produce bounds outside the feasible parameter range when the denominator is poorly estimated.

The NumPy/statsmodels example reports a pooled difference test and a finite Fieller-style interval for a ratio of independent sample proportions. The input counts are successes and sample sizes for A and B. It uses a normal approximation, not an exact small-sample binomial method, and deliberately rejects boundary proportions or a nonpositive denominator. Near-zero rates require a method suited to that regime; a finite output alone does not validate the approximation.

import numpy as np
from statsmodels.stats.proportion import proportions_ztest

def relative_uplift_interval(x_a, n_a, x_b, n_b, t=1.96):
    """Fieller interval for (CR_B - CR_A) / CR_A, as in Kohavi et al. (2009)."""
    if not (n_a > 0 and n_b > 0 and 0 < x_a < n_a and 0 < x_b < n_b):
        raise ValueError("example requires positive sizes and interior sample proportions")
    p_a, p_b = x_a / n_a, x_b / n_b
    # cv_a and cv_b are the coefficients of variation of the two rates.
    cv_a = np.sqrt(p_a * (1 - p_a) / n_a) / p_a
    cv_b = np.sqrt(p_b * (1 - p_b) / n_b) / p_b
    denominator = 1 - t**2 * cv_a**2
    if denominator <= 0:
        raise ValueError("the control rate is too uncertain for this interval")
    spread = t * np.sqrt(cv_a**2 + cv_b**2 - t**2 * cv_a**2 * cv_b**2)
    ratio = p_b / p_a
    return ratio * (1 - spread) / denominator - 1, ratio * (1 + spread) / denominator - 1

z, p = proportions_ztest([2650, 2500], [50000, 50000])
print(f"{z:.4f} {p:.4f}")
# 2.1462 0.0319

for arms in [(2500, 50000, 2650, 50000), (25, 500, 32, 500), (4, 80, 7, 80)]:
    low, high = relative_uplift_interval(*arms)
    print(f"{low:+.4f} {high:+.4f}")
# +0.0051 +0.1180
# -0.2384 +1.2360
# -0.5583 +38.5035

The last line is inside the domain by a hair and outside anything useful: an upper bound of 3,850 percent is the arithmetic reporting, correctly, that eighty visits say almost nothing about a 5 percent rate. The figure of 73 that appears above is the boundary for these particular observed rates under this normal approximation, not a general minimum sample size. A bootstrap interval is a practical alternative when the analytic form is awkward, and it inherits the same limitation rather than removing it.

Reading the result

• State both scales, or state the units. A relative figure without the baseline cannot be converted, and an absolute one without the baseline cannot be judged for size.
• Report the interval, not only the point. An estimated +6% relative uplift with an interval from +0.5% to +11.8% is compatible with materially different benefits. A launch depends on costs, risks, and the decision criteria; a forecast should represent this uncertainty rather than assume exactly 6%.
• Check the unit of analysis. A per-visit metric does not itself assume independence, but a test treating repeated visits as independent may understate uncertainty. Align the estimator with the desired per-user or per-visit quantity and account for dependence, for example with an appropriate cluster variance or user-level resampling.
• Fix the decision rule before looking. Stopping when the uplift first crosses the threshold inflates the error rate the test claims to control — see risk of peeking.
• Keep the campaign-level number separate from the per-customer one. State whether the target is assignment, exposure, or another defined population. A campaign average does not reveal each individual’s counterfactual outcome; conditional average effects are the subject of uplift and uplift models.

References

• Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne, "Controlled experiments on the web: survey and practical guide", Data Mining and Knowledge Discovery 18(1), 2009
• Nicholas J. Radcliffe and Patrick D. Surry, Real-World Uplift Modelling with Significance-Based Uplift Trees, Portrait Technical Report TR-2011-1, Stochastic Solutions White Paper, 2011