Cumulative Uplift

Cumulative uplift is an estimate of incremental outcome as a targeting policy goes deeper down a ranking. Customers are sorted by uplift score, highest first; at any depth, cumulative uplift answers the additional conversion, revenue or response estimated for the selected prefix under treatment rather than control. It is the quantity plotted on the vertical axis of an uplift curve, and the input to every summary built on that curve.

What is actually summed

The natural way to write it is a sum over customers of what the treatment did to each, and that expression cannot be evaluated. Each customer was either treated or not, so one of the two outcomes needed for the difference is missing for every row in the data. Radcliffe and Surry state the constraint in operational terms: “we cannot measure the uplift for an individual because we cannot simultaneously treat and not treat a single person”. A formula that sums per-customer differences is describing the quantity, not the computation.

What is computed instead is a comparison of two groups at each depth. Take the top \(n\) customers by score, of whom \(n_t\) were treated with \(n_{t1}\) responders and \(n_c\) were controls with \(n_{c1}\) responders. The cumulative uplift at that depth is the difference in response rates, scaled to the number of people targeted:

\[ \text{cumulative uplift}(n) = \left( \frac{n_{t1}}{n_t} - \frac{n_{c1}}{n_c} \right) \cdot n \]

The rates are not optional. Treatment allocation need not be equal, so \(n_t\) and \(n_c\) differ, and subtracting raw response counts answers a different question — often with the wrong sign at shallow depths. The distinction is worked through with numbers under uplift curve.

Here, \(n=n_t+n_c\) is the entire evaluation prefix, including its controls. Multiplying the rate difference by \(n\) estimates the gain from treating that whole prefix; it is not the number of conversions already caused among the \(n_t\) treated observations. For that treated-size scale, multiply by \(n_t\) instead. A causal interpretation requires a suitable experiment or adjustment, pre-treatment ranking features, and evaluation data not reused to fit the ranking. Both arms must be represented in each reported prefix.

Two things called cumulative uplift

The expression above produces a count: incremental conversions among the targeted. Drop the factor \(n\) and it produces a rate instead: the average effect per targeted customer, over everyone targeted so far. Both are reported as cumulative uplift, and they behave differently enough that a chart without an axis label is ambiguous.

causalml separates them in its code. Its get_cumlift returns “average uplifts of model estimates in cumulative population”, computed as the cumulative treated response rate minus the cumulative control response rate. Its get_cumgain then applies one line of arithmetic — cumulative gain is cumulative lift multiplied by the size of the population targeted — to convert the rate into the count. The rate is the natural thing to compare against a base rate; the count can be converted to value and compared with campaign cost.

A worked example

The data are synthetic: 20,000 customers, 30 percent of them treated, a base response rate of 5 percent, and a real effect of 6 percentage points confined to the top third of the ranking. Scoring is by the true ordering, which is optimal in expectation rather than a ceiling that no realized curve can cross. The table shows what that ordering delivers on this population.

import numpy as np

def make_population(seed=7, N=20000):
    """20,000 customers, 30 percent treated, a 5 percent base rate,
    and a 6 percentage point effect confined to the top third of the ranking."""
    rng = np.random.default_rng(seed)
    treatment = (rng.random(N) < 0.3).astype(int)
    score = rng.random(N)
    p = 0.05 + treatment * np.where(score > 2/3, 0.06, 0.0)
    return (rng.random(N) < p).astype(int), score, treatment

def cumulative(y, score, treatment, bins=10):
    """Running average uplift and running total, by decile of the ranking."""
    order = np.argsort(-score, kind="mergesort")
    y, treatment = y[order], treatment[order]
    n = len(y)
    rows = []
    previous_total = 0.0
    for b in range(1, bins + 1):
        k = n * b // bins
        t, c = treatment[:k], 1 - treatment[:k]
        rate_t = (y[:k] * t).sum() / t.sum()
        rate_c = (y[:k] * c).sum() / c.sum()
        # Cumulative uplift as a rate, then the same thing as a count.
        average = rate_t - rate_c
        total = average * k
        rows.append((b * 100 // bins, k, average, total, total - previous_total))
        previous_total = total
    return rows

y, score, treatment = make_population()
for depth, k, average, total, marginal in cumulative(y, score, treatment):
    print(f"{depth:>3}% {k:>6} {average:+.4f} {total:>8.1f} {marginal:>+8.1f}")
# 10% 2000 +0.0661 132.2 +132.2
# 20% 4000 +0.0484 193.6 +61.4
# 30% 6000 +0.0547 328.3 +134.7
# 40% 8000 +0.0482 385.5 +57.2
# 50% 10000 +0.0396 396.4 +10.9
# 60% 12000 +0.0327 391.9 -4.5
# 70% 14000 +0.0293 410.3 +18.4
# 80% 16000 +0.0251 401.9 -8.4
# 90% 18000 +0.0212 381.8 -20.1
# 100% 20000 +0.0182 363.3 -18.5

Depth	Targeted	Cumulative uplift, as a rate	Cumulative uplift, as a count	Change in the cumulative estimate
10%	2,000	+6.61 pp	132.2	+132.2
20%	4,000	+4.84 pp	193.6	+61.4
30%	6,000	+5.47 pp	328.3	+134.7
40%	8,000	+4.82 pp	385.5	+57.2
50%	10,000	+3.96 pp	396.4	+10.9
60%	12,000	+3.27 pp	391.9	-4.5
70%	14,000	+2.93 pp	410.3	+18.4
80%	16,000	+2.51 pp	401.9	-8.4
90%	18,000	+2.12 pp	381.8	-20.1
100%	20,000	+1.82 pp	363.3	-18.5

The two columns tell different stories. As a rate, cumulative uplift falls from 6.61 to 1.82 percentage points, with one bump at 30 percent, which is roughly what a working ranking should do: the true effect drops to zero below the top third, so including more of that region dilutes the average. The effects inside the top third are constant by construction, not strictly decreasing decile by decile. As a count it rises, wobbles, peaks at 410 around 70 percent depth, and ends lower than it peaked. Neither column is wrong; they answer different questions, and the count must still be converted to contribution or revenue before comparing it with a monetary budget.

The last column is the one to read when deciding how deep to go. It is the change in the cumulative estimate from one depth to the next, which is not the same as the effect estimated on that decile alone: enlarging the prefix reweights everything already in it. Radcliffe and Surry’s case makes the size of that clear — two segments estimated at 60 and 20 conversions separately can pool to 128, so reading 128 - 60 = 68 as the second segment’s contribution overstates it threefold. Past the top third the column is small and unstable — the population below the top third has a true effect of exactly zero here, so those entries are noise around nothing. A negative entry does not mean the decile contains customers the treatment harms; in this data it cannot, because no such customers exist.

Reading a downturn

When the count peaks before the right-hand edge, two explanations compete and the chart cannot distinguish them. One is a genuine negative effect. Radcliffe and Surry show a model whose curve is higher at 80 percent volume than at 100 percent, “because of negative effects in the last two deciles” — customers the treatment puts off. The other is sampling noise, which is what the table above shows. Estimating a difference of rates is noisier than estimating either rate, and the segments at the bottom of a ranking are where the estimate is weakest relative to the effect.

Telling them apart requires more than the curve: the size of the segments driving the downturn, the interval around their realized uplift, and whether the same pattern appears in a fresh sample. Radcliffe and Surry list monotonicity of incremental gains across reasonably large segments among the checks they run alongside any summary metric, and the emphasis on reasonably large is the whole content of the check.

Where to stop

With positive treatment cost, maximizing the count of incremental outcomes can spend more than maximizing profit. Convert the expected gain at each candidate depth to contribution and subtract treatment cost. Under a smoothly declining marginal effect and constant positive cost, the profit maximum occurs before the gain curve flattens. Noisy empirical curves, varying costs, and capacity constraints require comparing feasible policies directly.

Radcliffe and Surry derive the cut-off from the curve by applying cost and value. With treatment costing \$1,000 per 1 percent of the population and each incremental sale contributing \$150, one of their models reaches peak profit at 40 percent volume — 680 incremental sales and about \$62,000 — and another at 60 percent, with 870 sales and about \$70,500. Both cut-offs sit well before the point where either curve stops rising. The second model is worse at shallow depths and better overall, which is only visible once money is attached to the axis.

How it relates to the other measures

Quantity	What it is
Cumulative uplift	Incremental outcome among everyone targeted down to a depth, as a rate or as a count
Marginal, or per-bucket, incremental gain	The difference between consecutive cumulative values. Read it as the change in the estimate, not as the effect of that decile on its own — for a standalone segment figure, estimate the segment separately, and for a decomposition that adds up, use fixed weights
Uplift@k	The cumulative uplift rate at one chosen depth, reported as a single number
Uplift curve	Cumulative uplift as a count, plotted against depth
Qini curve	The same idea with the control arm rescaled by the treated-to-control ratio instead, so its vertical values are not cumulative uplift
AUUC and the Qini coefficient	Areas under those curves, reduced to one number

At a fixed depth, pooling the same prefix produces the same estimate regardless of the chart’s bin count. Adding separately estimated segment gains can differ because each segment uses its own treated and control denominators. This concerns the estimator, not non-additivity of causal effects: population average effects combine using common population weights. If an additive report is needed, fix the target population and weighting convention explicitly, and distinguish that estimate from the raw pooled difference.

References

• Nicholas J. Radcliffe and Patrick D. Surry, Real-World Uplift Modelling with Significance-Based Uplift Trees, Portrait Technical Report TR-2011-1, Stochastic Solutions White Paper, 2011
• causalml source, causalml/metrics/visualize.py (get_cumlift, get_cumgain, get_qini)
• scikit-uplift 0.5.1 source, sklift/metrics/metrics.py (uplift_curve)