Incremental Gain

Incremental gain is the extra outcome a treatment produced inside one segment of a population — one decile of a ranking, one customer tier, one campaign cell. It is the local, step-by-step version of the quantity that cumulative uplift accumulates, and the building block of every curve and summary in uplift evaluation.

What a segment’s gain is

For a segment containing \(n\) customers, of whom \(n_t\) were treated with \(n_{t1}\) responders and \(n_c\) were controls with \(n_{c1}\) responders, the incremental gain is

\[ \text{gain} = \left( \frac{n_{t1}}{n_t} - \frac{n_{c1}}{n_c} \right) \cdot n \]

The difference of rates, scaled to the number of people in the segment. Radcliffe and Surry state the condition that makes such a comparison legitimate: uplift can be estimated for a segment “provided that we take equivalent subpopulations in each of the treated and control and that those subpopulations are large enough to be meaningful”. Equivalent means drawn from the same place in the ranking, not equal in size — and that is where the arithmetic usually goes wrong.

Why counts give the wrong answer

The intuitive formula is responders in the treated group minus responders in the control group. It answers a narrower question than it looks: the raw difference estimates the effect scaled to one arm’s size, and it coincides with the scaled form only when \(N\) is taken to be that size and the arms are equal. A holdout is typically a minority of the population, so with unequal arms the raw difference is not even that — the control arm books fewer responses simply by being smaller.

Equal arms do not rescue the raw difference either, because the two forms answer to different \(N\). With 100 customers per arm converting 20 and 10, the raw difference is 10 and the scaled form with \(N = n_t + n_c = 200\) gives 20. Both are estimates, and neither is a count of what the campaign caused: the control arm’s 10 conversions are not the treated arm’s own counterfactual, only a stand-in for it that randomization makes unbiased on average. In any single experiment the realized effect on the treated can differ — 10 is perfectly consistent with a true effect of zero.

So the 10 estimates the effect scaled to one arm’s size, the 20 estimates it for a population of 200, and \(N\) has to be fixed before either number means anything: one arm’s size, the tested population, or the rollout audience for a projection that additionally assumes the effect carries over.

The table below makes the size of the error concrete. The data are synthetic — 20,000 customers, 30 percent treated, a 5 percent base rate, and a real effect of 6 percentage points confined to the top third of the ranking.

import numpy as np

def make_population(seed=7, N=20000):
    """20,000 customers, 30 percent treated, a 5 percent base rate,
    and a 6 percentage point effect confined to the top third of the ranking."""
    rng = np.random.default_rng(seed)
    treatment = (rng.random(N) < 0.3).astype(int)
    score = rng.random(N)
    p = 0.05 + treatment * np.where(score > 2/3, 0.06, 0.0)
    return (rng.random(N) < p).astype(int), score, treatment

def gain_by_decile(y, score, treatment, bins=10):
    """Incremental gain within each decile of the ranking, the wrong way and the right way."""
    order = np.argsort(-score, kind="mergesort")
    y, treatment = y[order], treatment[order]
    for chunk_y, chunk_t in zip(np.array_split(y, bins), np.array_split(treatment, bins)):
        n = len(chunk_y)
        n_t, n_c = chunk_t.sum(), n - chunk_t.sum()
        y_t, y_c = chunk_y[chunk_t == 1].sum(), chunk_y[chunk_t == 0].sum()
        raw = y_t - y_c
        gain = (y_t / n_t - y_c / n_c) * n
        yield n_t, n_c, y_t, y_c, raw, gain

y, score, treatment = make_population()
rows = list(gain_by_decile(y, score, treatment))
for n_t, n_c, y_t, y_c, raw, gain in rows:
    print(f"{n_t:>5} {n_c:>5} {y_t:>4} {y_c:>4} {raw:>+5} {gain:>+8.1f}")
# 589 1411 69 72 -3 +132.2
# 592 1408 51 78 -27 +61.5
# 623 1377 75 74 +1 +133.3
# 607 1393 48 70 -22 +57.7
# 637 1363 26 44 -18 +17.1
# 596 1404 24 61 -37 -6.4
# 613 1387 31 57 -26 +19.0
# 569 1431 30 83 -53 -10.6
# 612 1388 28 77 -49 -19.4
# 567 1433 21 69 -48 -22.2

total = sum(r[5] for r in rows)
# The deciles do not add up to the figure for the whole population.
n_t, n_c = treatment.sum(), (1 - treatment).sum()
y_t, y_c = y[treatment == 1].sum(), y[treatment == 0].sum()
whole = (y_t / n_t - y_c / n_c) * len(y)
print(f"{total:.1f} {whole:.1f} {total - whole:+.1f}")
# 362.1 363.3 -1.2

Decile	Treated	Controls	Treated responders	Control responders	Counts subtracted	Incremental gain
1	589	1,411	69	72	-3	+132.2
2	592	1,408	51	78	-27	+61.5
3	623	1,377	75	74	+1	+133.3
4	607	1,393	48	70	-22	+57.7
5	637	1,363	26	44	-18	+17.1
6	596	1,404	24	61	-37	-6.4
7	613	1,387	31	57	-26	+19.0
8	569	1,431	30	83	-53	-10.6
9	612	1,388	28	77	-49	-19.4
10	567	1,433	21	69	-48	-22.2

The counts column is negative in nine deciles out of ten, and it would have every one of them reported as a loss. The gain column is a set of finite-sample estimates. The generator gives a nonzero effect only in the top third: the fourth decile mixes customers with an effect and customers with zero effect. Below the top third the true effect is zero by construction, while observed estimates can be positive or negative. The two columns do not merely differ in scale; they differ in sign almost everywhere, because there are roughly two and a half times as many control customers as treated ones and the raw response-count difference can therefore be negative even when the treatment raises the response rate.

Raw segment gains do not add up

Summing the ten raw decile gains above gives 362.1 incremental responses. Computing the same quantity directly on the whole population gives 363.3. Neither is a rounding error, and neither is wrong.

Radcliffe and Surry give the phenomenon its own section and a worked case: one segment with 19,950 treated customers making 384 sales and 10,050 controls making 122 has an uplift of 0.711 percentage points, a second with 20,050 treated making 464 and 9,950 controls making 202 has 0.284 points, their average is 0.497, and the combined population gives 0.500. Their note is worth quoting in full: “This is not the result of rounding error.” The cause is that the treated share is not identical in the two segments, which makes this a relative of Simpson’s paradox rather than an arithmetic slip.

The practical consequence is that a decile table and a percentile table of the same campaign, each built from raw segment estimates, will not agree exactly, and these estimates cannot be reconciled to the last decimal place. Report one of them, say which, and do not present the sum of the raw parts as the total. A decomposition that does add up defines every slice with the same fixed population weights, and cumulative incremental gain avoids the problem by comparing the treated and control rates directly on each selected prefix.

Reading a negative segment

Deciles 6, 8, 9 and 10 in the table show negative gains. It is tempting to read them as evidence that the treatment harms those customers — the sleeping dogs of the marketing literature. In this data that reading is provably wrong: the bottom two-thirds of the population have a true effect of exactly zero by construction, and the negative numbers are noise around it.

Nothing on the surface distinguishes that case from a real negative effect, which is why a negative segment gain is a hypothesis rather than a finding. What separates them is the size of the segment, the interval around its estimate, and whether the pattern repeats on data the model has not seen. A difference of two rates has a larger standard error than either rate, and a decile is a small enough slice for that to dominate — see confidence intervals. Radcliffe and Surry’s own check is monotonicity of incremental gains across “reasonably large segments”, and the qualification is doing most of the work.

Where it is used

• Finding where the effect lives. A gain concentrated in the top deciles is the pattern a targeting model exists to produce, and a gain spread evenly down the ranking means the ordering carries no information about the effect.
• Choosing a depth. The marginal gain of the next slice is what a volume decision needs, but the profitable cut-off is not where the gain reaches zero — it is where the gain stops covering the cost of treating that slice. Cumulative uplift works the arithmetic through.
• Building the curve. Computing the treated–control rate comparison directly on each selected prefix produces cumulative incremental gain, which is what the uplift curve plots and what total incremental benefit reports at the end.

One assumption runs under all of it. The comparison inside a segment is causal because assignment to treatment was random; on data where an earlier campaign chose its own recipients, the unadjusted segment gain measures that choice as faithfully as it measures the treatment — recovering the effect there needs propensity or outcome adjustment and the assumptions that go with it. The baseline the whole exercise is read against is covered under random targeting strategy.

References

• Nicholas J. Radcliffe and Patrick D. Surry, Real-World Uplift Modelling with Significance-Based Uplift Trees, Portrait Technical Report TR-2011-1, Stochastic Solutions White Paper, 2011
• scikit-uplift 0.5.1 source, sklift/metrics/metrics.py (uplift_curve)