Uplift@k

Uplift@k is the incremental effect measured among the top \(k\) fraction of a population ranked by predicted uplift score. Where a curve shows every possible targeting depth, uplift@k picks one and reports a single number, which is what a campaign with a fixed volume actually needs. Writing \(y^T_{(k)}\) and \(y^C_{(k)}\) for the responders and \(n^T_{(k)}\), \(n^C_{(k)}\) for the counts in each arm within that top slice,

\[ \text{uplift@}k = \frac{y^T_{(k)}}{n^T_{(k)}} - \frac{y^C_{(k)}}{n^C_{(k)}} \]

It is a difference of response rates, read in percentage points, and it is the same quantity that cumulative uplift reports as a rate — taken at one depth rather than traced across all of them.

What it is for

Most summary metrics for uplift models average over targeting depths, including depths no campaign will use. Uplift@k does not. If the budget covers 20 percent of the base, the number that decides between two models is what each delivers at 20 percent, and Radcliffe and Surry make that the first thing they check when a cut-off is predetermined: one of their models delivers an estimated 450 incremental sales at that volume against another’s 380, while above 40 percent the ordering reverses. Neither model dominates, and a single area-based score would have hidden the crossing.

The corollary is that uplift@k without its \(k\) cannot be read. A model reporting 0.066 at 10 percent and 0.040 at 50 percent is one model, not two, and the decline is the normal shape of a working ranking rather than a defect.

A worked example

The data are synthetic: 20,000 customers, 30 percent of them treated, a base response rate of 5 percent, and a real effect of 6 percentage points confined to the top third of the ranking.

import numpy as np

def make_population(seed=7, N=20000):
    """20,000 customers, 30 percent treated, a 5 percent base rate,
    and a 6 percentage point effect confined to the top third of the ranking."""
    rng = np.random.default_rng(seed)
    treatment = (rng.random(N) < 0.3).astype(int)
    score = rng.random(N)
    p = 0.05 + treatment * np.where(score > 2/3, 0.06, 0.0)
    return (rng.random(N) < p).astype(int), score, treatment

def make_drifting_population(seed=7, N=20000):
    """The same effect, but the treated share falls from about 50 percent at the
    top of the ranking to about 10 percent at the bottom, as it does when the
    previous campaign chose its own recipients."""
    rng = np.random.default_rng(seed)
    score = rng.random(N)
    treatment = (rng.random(N) < 0.1 + 0.4 * score).astype(int)
    p = 0.05 + treatment * np.where(score > 2/3, 0.06, 0.0)
    return (rng.random(N) < p).astype(int), score, treatment

def uplift_at_k(y, score, treatment, k, strategy="overall"):
    order = np.argsort(score, kind="mergesort")[::-1]
    y, treatment = y[order], treatment[order]
    if strategy == "overall":
        n = int(len(y) * k)
        top_y, top_t = y[:n], treatment[:n]
        return top_y[top_t == 1].mean() - top_y[top_t == 0].mean()
    n_t = int((treatment == 1).sum() * k)
    n_c = int((treatment == 0).sum() * k)
    return y[treatment == 1][:n_t].mean() - y[treatment == 0][:n_c].mean()

def standard_error(y, score, treatment, k):
    order = np.argsort(score, kind="mergesort")[::-1]
    y, treatment = y[order], treatment[order]
    n = int(len(y) * k)
    a = y[:n][treatment[:n] == 1]
    b = y[:n][treatment[:n] == 0]
    return float(np.sqrt(a.mean() * (1 - a.mean()) / len(a)
                         + b.mean() * (1 - b.mean()) / len(b)))

y, score, treatment = make_population()
for k in (0.1, 0.2, 0.5, 0.999):
    print(f"{k:.3f} {uplift_at_k(y, score, treatment, k):+.4f}")
# 0.100 +0.0661
# 0.200 +0.0484
# 0.500 +0.0396
# 0.999 +0.0182

useless = np.random.default_rng(999).random(len(y))
# A ranking that carries no information does not give zero; it estimates the
# whole-population effect, which is the value at k = 100 percent.
print(f"{uplift_at_k(y, useless, treatment, 0.2):+.4f}")
# +0.0178

point = uplift_at_k(y, score, treatment, 0.2)
se = standard_error(y, score, treatment, 0.2)
print(f"{point:+.4f} {point - 1.96 * se:+.4f} {point + 1.96 * se:+.4f}")
# +0.0484 +0.0293 +0.0675

yd, sd, td = make_drifting_population()
# The two strategies agree while the treated share is flat, and part once it drifts.
for k in (0.2, 0.5):
    print(f"{k:.1f} {uplift_at_k(yd, sd, td, k, 'overall'):+.4f} "
          f"{uplift_at_k(yd, sd, td, k, 'by_group'):+.4f}")
# 0.2 +0.0646 +0.0730
# 0.5 +0.0490 +0.0670

k	Uplift@k	What it means
10%	+6.61 pp	The top decile is mostly made of customers the treatment moves
20%	+4.84 pp	Still well above the population average
50%	+3.96 pp	About one third of the targeted group has zero true effect in this construction
Essentially everyone	+1.82 pp	The average effect over the whole population

The baseline is not zero

It is tempting to read uplift@k as a measure that a useless model would score near zero on. It is not. A ranking that carries no information about the effect still selects a random slice of the population, and a random slice has the population’s average effect. In the example above, scoring the same customers with random numbers gives an uplift@20% of +1.78 percentage points, against +1.82 for the population as a whole — the two agree to within sampling error, and neither is zero.

So uplift@k mixes two things: whether the treatment works at all, and whether the model finds the people it works on. The comparison that isolates the second is uplift@k against the value at \(k = 100\) percent. A result no higher than the whole-population estimate shows no targeting advantage at that depth in this sample. It does not establish that the model learned nothing: compare joint uncertainty, other relevant depths, and independent evaluation data before concluding that effect ordering is absent.

Its uncertainty

Uplift@k is a difference between two observed proportions, so it has the standard error of one: \(\sqrt{\hat{p}_T(1 - \hat{p}_T)/n^T_{(k)} + \hat{p}_C(1 - \hat{p}_C)/n^C_{(k)}}\). That interval describes one model. Comparing two models on the same evaluation set is a different calculation: the two estimates share the data and are correlated, so the difference has standard error \(\sqrt{\operatorname{Var}(A) + \operatorname{Var}(B) - 2\operatorname{Cov}(A,B)}\), which can be far smaller than either interval suggests — with each standard error 0.01 and a correlation of 0.99, the difference has a standard error of 0.0014, and a gap of 0.5 points is more than three standard errors. Use a paired bootstrap on the same units rather than reading two separate intervals. In the example, uplift@20% is +4.84 percentage points with a 95 percent interval running from +2.93 to +6.75 — a range of nearly four points on a number quoted to two decimal places. That width is the reason not to over-read a single figure. It is not a verdict on two models differing by half a point at the same \(k\): whether that gap is real is the paired calculation above, and with correlated estimates it can be significant while both intervals overlap almost completely.

The interval also narrows more slowly than intuition suggests as \(k\) grows, because the slice is split between two arms and a holdout is usually the smaller one. At shallow \(k\) on a modest holdout the control arm can be a few hundred rows, and the metric is then reporting mostly noise. Confidence intervals and two-proportion z-test cover the arithmetic.

Which top k

There are two ways to take the top \(k\), and they are not equivalent. scikit-uplift exposes both through a strategy argument. Under overall, the first \(k\) of the whole ordered sample is taken and the response rates of its treated and control members are compared. Under by_group, the top \(k\) of the treated and the top \(k\) of the controls are taken separately and then compared.

While the treated share is roughly constant down the ranking the two agree closely: at \(k = 20\) percent the example gives +4.84 against +4.92 points. They part company when the share drifts, which is exactly what happens on data from a campaign that chose its own recipients. Rerunning with a treated share falling from about 50 percent at the top of the ranking to about 10 percent at the bottom gives +4.90 points under overall and +6.70 under by_group at \(k = 50\) percent, a difference of 37 percent in the reported effect.

Neither mode removes selection on its own. Taking the same prefix for both arms fixes which cases are compared, not who was treated within them: if a high-baseline stratum is 90 percent treated and a low-baseline stratum 10 percent treated, the prefix difference can read +0.8 when every true effect is zero. Constant-probability randomization, or weighting and standardization, is what makes either number causal. The reason the two modes differ here is visible once the two slices are inspected. Under by_group at that depth, the top half of the treated reaches down only to a score of 0.667, while the top half of the controls reaches to 0.424: every customer in the treated slice is one the treatment can move, against 54 percent of those in the control slice. The comparison is between two different populations, and it flatters the model. The overall strategy draws both arms from one prefix and does not have this problem, which makes it the safer default — and makes the strategy a thing to record next to the number.

Where it sits

Question	What answers it
Does the treatment work on this population?	Uplift over everyone, which is uplift@k at \(k\) of 100 percent
Is the model worth using at the volume we send?	Uplift@k at that volume, against the whole-population value
What volume should we send?	The uplift curve with cost and value applied to it
Which of two models ranks better overall?	The Qini coefficient or AUUC, read with their normalization stated

One caution carries over from every measure in this family. Uplift@k is a segment estimate, and segment uplift estimates are not additive, so segment values cannot simply be added to reach a prefix value; they combine only as a weighted average under common population weights. At the same cutoff the pooled prefix gives the same number at either resolution; what differs is which cutoffs a decile grid and a percentile grid make available. Randomization buys conditional exchangeability and overlap; no interference between customers is a separate design requirement. And the raw comparison inside the slice is causal here because assignment was randomized; on data where a previous rule chose the treated, the raw uplift@k measures that rule as faithfully as it measures the model, unless the weighting or standardization described above rests on identification assumptions that hold.

References

• scikit-uplift 0.5.1 documentation, sklift.metrics.uplift_at_k
• scikit-uplift 0.5.1 source, sklift/metrics/metrics.py (uplift_at_k)
• Nicholas J. Radcliffe and Patrick D. Surry, Real-World Uplift Modelling with Significance-Based Uplift Trees, Portrait Technical Report TR-2011-1, Stochastic Solutions White Paper, 2011