Uplift Curve

An uplift curve estimates how incremental outcome accumulates under a model’s targeting order as the targeted share of the population grows. Customers are ranked by predicted uplift, highest first; the horizontal axis is the proportion targeted and the vertical axis is an estimate of the cumulative incremental response among those targeted. Radcliffe and Surry, who call it an incremental gains curve, describe the construction exactly: people are “chosen in descending order of quality as ranked by the model in question”, and the vertical value is “produced by comparing the cumulative purchase rate, targeting by model score, in the treated and control populations”. The diagonal from the origin to the endpoint is a reference for expected uniform random targeting at each depth using the estimated full-population effect; a realized random targeting curve can fluctuate around it.

The curve exists because an uplift model cannot be scored the way a classifier is. The quantity it predicts is a difference between two outcomes, only one of which happens to any customer, so there is no per-row label to compare a prediction against. What can be examined is whether the ranking separates realized outcomes in aggregate, and that is a question about groups rather than individuals. The curve answers it at every possible cut-off at once.

How each point is computed

Take the top \(n\) customers by predicted uplift, of whom \(n_t\) were treated and \(n_c\) were controls, with \(n_{t1}\) and \(n_{c1}\) responders in each. The uplift curve’s value at that depth is the difference in response rates, scaled back up to the number of people targeted:

\[ \text{uplift}(n) = \left( \frac{n_{t1}}{n_t} - \frac{n_{c1}}{n_c} \right) \cdot n \]

The rates matter because \(n_t\) and \(n_c\) are not equal, and need not even be in the same proportion at every depth. A holdout is usually a minority of the population, and the ranking is free to sort treated and control customers unevenly. Subtracting raw response counts — \(n_{t1} - n_{c1}\) — answers a different question, and at a shallow depth it usually answers it wrongly.

The closely related Qini curve rescales the control responders instead of the rates, \(n_{t1} - n_{c1} \cdot n_t / n_c\), which estimates treated responders minus a control-based counterfactual count. The causal reading requires comparable groups under the evaluation design. At a prefix containing both arms, the uplift value equals the Qini value multiplied by \(n / n_t\), the reciprocal of the treated share at that depth. When the treated share is exactly one half at every evaluated prefix, the uplift curve is twice the Qini curve; with a drifting treated share the two differ in shape as well as in height, which is one reason a number read off one is not comparable with a number read off the other.

A worked example

The data below are synthetic. Twenty thousand customers, 30 percent of them treated, a base response rate of 5 percent, and a real effect of 6 percentage points confined to the top third of the ranking. The model’s score is the true ordering, so it is optimal in expectation. That is not the same as an upper bound on the realized curve: the effect is a constant 6 points inside the top third and zero outside it, so any reordering within those blocks has the same expected value, and sampling variation means some other ranking can sit higher at a given depth or over the whole area without being a better policy. So read the curve below as what an ideal ordering delivers in expectation on this population, not as a ceiling every other ranking must fall under. The NumPy implementation sorts by score, evaluates complete tied-score groups, and maintains cumulative responder and arm counts. Its numerical convention follows the cited scikit-uplift definitions. At a prefix missing one arm, clamping the denominator avoids division by zero but does not create evidence for the missing response rate; do not interpret that point as an identified effect.

import numpy as np

def make_population(seed=7, N=20000):
    """20,000 customers, 30 percent treated, a 5 percent base rate,
    and a 6 percentage point effect confined to the top third of the ranking."""
    rng = np.random.default_rng(seed)
    treatment = (rng.random(N) < 0.3).astype(int)
    score = rng.random(N)
    p = 0.05 + treatment * np.where(score > 2/3, 0.06, 0.0)
    return (rng.random(N) < p).astype(int), score, treatment

def curves(y, uplift, treatment):
    """Uplift and Qini curve values at every cut point of the ranking."""
    order = np.argsort(uplift, kind="mergesort")[::-1]
    y, uplift, treatment = y[order], uplift[order], treatment[order]
    cut = np.r_[np.where(np.diff(uplift))[0], y.size - 1]
    # n customers targeted, of whom n_t treated and n_c control;
    # y_t and y_c are the responders in each arm.
    n = cut + 1
    n_t = np.cumsum(treatment)[cut]
    n_c = n - n_t
    y_t = np.cumsum(y * treatment)[cut]
    y_c = np.cumsum(y * (1 - treatment))[cut]
    # The shallowest cut points have one arm empty; clamp so the division is defined.
    t, c = np.maximum(n_t, 1), np.maximum(n_c, 1)
    return (np.r_[0, n],
            np.r_[0.0, (y_t / t - y_c / c) * n],
            np.r_[0.0, y_t - y_c * n_t / c])

y, score, treatment = make_population()
x, uplift_curve, qini_curve = curves(y, score, treatment)

for depth in (0.1, 0.5, 1.0):
    i = np.searchsorted(x, round(len(y) * depth))
    print(f"{depth:.0%} {x[i]} {uplift_curve[i]:.2f} {qini_curve[i]:.2f}")
# 10% 2000 132.24 38.94
# 50% 10000 396.35 120.81
# 100% 20000 363.29 109.08

The first row is the warning. Among the top 2,000 customers there are 69 treated responders and 72 control responders, so the raw difference in counts is -3 and a naive chart would draw the best decile below zero. But those 69 came from 589 treated customers and those 72 from 1,411 controls, giving rates of 11.7 percent against 5.1 percent, and the curve’s value is \((0.117 - 0.051) \times 2000 = 132\). The same arithmetic on the full population reproduces the endpoints: \((403/6005 - 685/13995) \times 20000 = 363.29\) for the uplift curve and \(403 - 685 \times 6005/13995 = 109.08\) for the Qini curve. The ratio of the two columns is \(n/n_t\) throughout, between 3.33 and 3.40 here because roughly 30 percent of each prefix is treated.

Reading the curve

A steep start estimates high cumulative incremental response near the top. A curve near the diagonal may indicate little useful effect ordering, but sampling variation and the evaluation design limit that inference. Assess independent evaluation data and uncertainty before concluding that a ranking has or lacks effect information. The height at a chosen depth is the decision-relevant number when the campaign volume is fixed: in Radcliffe and Surry’s illustration one model delivers an estimated 450 incremental sales at 20 percent volume against another’s 380, and above 40 percent the ordering reverses — neither model dominates, and which is better is a question about the volume you intend to send.

Where to cut is a question about money rather than about the curve. Applying a cost per contact and a value per incremental sale turns the same curve into a profit curve. With their figures — $1,000 to treat each 1 percent of the population and $150 of contribution per incremental sale — the first model’s profit peaks at 40 percent volume with 680 incremental sales and about $62,000, and the second’s at 60 percent with 870 sales and about $70,500. The model that looked worse at shallow depths is the one that makes more money, because the campaign it supports is larger.

A curve that turns down before the right-hand edge is worth pausing over, because two very different things produce that shape. One is a genuine negative effect: in Radcliffe and Surry’s figure, targeting 80 percent of the population yields more incremental sales than targeting all of it, because the last two deciles contain customers the treatment puts off. The other is noise. In the table above the curve rises to 410 at 70 percent depth and falls to 363 by the end, and there are no negative effects in the data at all — the bottom two-thirds have a true effect of exactly zero, and sampling variation can move the estimated prefix gain up or down. Before reading a downturn as sleeping dogs, check whether the segments driving it are large enough for their estimates to mean anything, and whether the pattern survives in a fresh sample.

Why the buckets matter

Every point uses a prefix of the ranked sample. Averaging separate segment rate differences need not reproduce the pooled prefix difference when treated and control shares differ across segments. Radcliffe and Surry give a table where the arithmetic is plain. Splitting a population at the median model score gives one segment with 19,950 treated customers making 384 sales (1.925 percent) against 10,050 controls making 122 (1.214 percent), an uplift of 0.711 percentage points, and a second with 20,050 treated making 464 (2.314 percent) against 9,950 controls making 202 (2.030 percent), an uplift of 0.284 points. The average of the two is 0.497 points. The whole population, 40,000 treated making 848 against 20,000 controls making 324, gives 2.120 percent against 1.620 percent — an uplift of 0.500 points. Their note is worth keeping: “This is not the result of rounding error.” The gap comes from the treated share differing slightly between the segments, a relative of Simpson’s paradox.

Two practical consequences follow. At a given cutoff the number is fixed: the same prefix, summed by arm, gives the same value whether you reached it in ten steps or a hundred. What the resolution changes is where the curve is sampled, how the area between samples is interpolated, and — when separate per-bucket averages are combined rather than a single prefix being pooled — the arithmetic itself, because the two arms use different segment weights. Population causal effects, in contrast, average across disjoint segments using their common target-population shares. And where a curve is built from independently estimated buckets rather than from pooled prefixes, many small buckets are noisier than a few large ones without being more accurate — the extra resolution is resolution in the noise. That is a property of per-bucket estimation, not of the cumulative prefix, which is unchanged.

What the curve does not tell you

• Whether the model is calibrated. The curve depends only on the ranking. A model whose predicted uplifts are all wrong by a factor of ten draws the same curve as one that is right, and the two differ exactly where a cut-off is chosen from predicted values.
• Whether the pattern is real. For independent treatment and control rate estimates, the variance of the difference is the sum of their variances. Other designs or adjusted estimators need their own variance calculation. A curve without confidence intervals, or without a companion curve on held-out data, is one sample’s shape.
• Whether the comparison is causal. The treated and control groups are compared as they fall in the ranking. Without random assignment or justified adjustment, the raw curve can confound the treatment effect with selection into treatment. Observational evaluation requires explicit identification assumptions and a corresponding adjusted estimator. See A/B testing.
• A single number. Summarizing the area between the curve and the diagonal is the job of the Qini coefficient, and the choice of what to divide that area by changes the answer by more than most model differences do.

References

• Nicholas J. Radcliffe and Patrick D. Surry, Real-World Uplift Modelling with Significance-Based Uplift Trees, Portrait Technical Report TR-2011-1, Stochastic Solutions White Paper, 2011
• scikit-uplift 0.5.1 source, sklift/metrics/metrics.py (uplift_curve, qini_curve)
• Floris Devriendt, Tias Guns and Wouter Verbeke, Learning to rank for uplift modeling, arXiv:2002.05897, 2020