Total Incremental Benefit (TIB)

Total incremental benefit is everything a treatment produced that would not have happened anyway, across the whole population it was applied to. It is the value at full depth of cumulative incremental gain — the right-hand end of an uplift curve — and the figure a campaign post-mortem usually leads with.

What is counted

The quantity is a sum over people of what the treatment did to each, and that sum cannot be computed from any dataset: each person was either treated or not, so one of the two outcomes is always missing. Radcliffe and Surry state the constraint directly — “we cannot measure the uplift for an individual because we cannot simultaneously treat and not treat a single person”.

What is computed is a comparison of the two arms, scaled to the population that was exposed. With \(n_t\) treated customers producing \(n_{t1}\) responses and \(n_c\) controls producing \(n_{c1}\), over a population of \(N\),

\[ \text{TIB} = \left( \frac{n_{t1}}{n_t} - \frac{n_{c1}}{n_c} \right) \cdot N \]

Subtracting the raw response counts answers a different question from the scaled formula — it estimates the effect scaled to one arm’s size rather than to \(N\) — and with unequal arms it is not even that, since the smaller arm books fewer responses regardless of the campaign. The general case, and how badly counts mislead, is worked through under incremental gain.

Equal arms do not rescue the raw difference either, because the two forms answer to different \(N\). With 100 customers per arm converting 20 and 10, the raw difference is 10 and the scaled form with \(N = n_t + n_c = 200\) gives 20.

Both are estimates, and neither is a count of what the campaign caused: the control arm’s 10 conversions are not the treated arm’s own counterfactual, only a stand-in for it that randomization makes unbiased on average. In any single experiment the realized effect on the treated can differ — 10 is perfectly consistent with a true effect of zero. So the 10 estimates the effect scaled to one arm’s size, the 20 estimates it for a population of 200, and \(N\) has to be fixed before either number means anything: one arm’s size, the tested population, or the rollout audience for a projection that additionally assumes the effect carries over.

Turning it into money

The count becomes a business figure by multiplying by what a conversion is worth, and becomes a decision by subtracting what the treatment cost. The data below are synthetic: 20,000 customers, 30 percent treated, a 5 percent base response rate, a real effect of 6 percentage points confined to the top third of the ranking, a contribution of \$50 per incremental conversion, and \$2 to contact one customer.

import numpy as np

def make_population(seed=7, N=20000):
    """20,000 customers, 30 percent treated, a 5 percent base rate,
    and a 6 percentage point effect confined to the top third of the ranking."""
    rng = np.random.default_rng(seed)
    treatment = (rng.random(N) < 0.3).astype(int)
    score = rng.random(N)
    p = 0.05 + treatment * np.where(score > 2/3, 0.06, 0.0)
    return (rng.random(N) < p).astype(int), score, treatment

y, score, treatment = make_population()
n_t, n_c = treatment.sum(), (1 - treatment).sum()
p_t = y[treatment == 1].mean()
p_c = y[treatment == 0].mean()
N = len(y)

rate = p_t - p_c
se_rate = np.sqrt(p_t * (1 - p_t) / n_t + p_c * (1 - p_c) / n_c)
low, high = (rate - 1.96 * se_rate) * N, (rate + 1.96 * se_rate) * N
print(f"{rate * 100:.3f} {rate * N:.1f} {low:.1f} {high:.1f}")
# 1.816 363.3 217.9 508.7

margin, cost_per_contact = 50.0, 2.0
benefit = rate * N * margin
spend = N * cost_per_contact
print(f"{benefit:.0f} {spend:.0f} {benefit - spend:.0f} {(benefit - spend) / spend:.3f}")
# 18165 40000 -21835 -0.546

k = N // 3
# Treating only the top third instead of everyone.
order = np.argsort(-score, kind="mergesort")
top_y, top_t = y[order][:k], treatment[order][:k]
top_rate = top_y[top_t == 1].mean() - top_y[top_t == 0].mean()
benefit_top = top_rate * k * margin
spend_top = k * cost_per_contact
print(f"{top_rate * 100:.3f} {top_rate * k:.1f} {benefit_top:.0f} {spend_top:.0f} "
      f"{benefit_top - spend_top:.0f} {(benefit_top - spend_top) / spend_top:.3f}")
# 5.596 373.0 18651 13332 5319 0.399

The left column is a total incremental benefit of \$18,165 on a campaign that lost \$21,835. Nothing in the figure is wrong; it simply is not the number the decision needs, because it counts what the treatment added and ignores what it cost to add it. Naming it a benefit invites exactly that reading, which is why a reported total should always arrive next to the spend that produced it.

Why the total is not the maximum

The right-hand column treats a third as many customers and produces essentially the same number of incremental conversions — the excess over the full-population figure is noise, since the bottom two-thirds of this population have a true effect of exactly zero. The same outcome for a third of the cost turns a loss into a profit.

So the value at full depth is the result of one choice among many, not a ceiling. Radcliffe and Surry make the point with real campaigns: “Given cost and value information, we can determine the optimal cutoff for each model and choose the one that leads to the highest predicted campaign profit.” At \$1,000 to treat each 1 percent of the population and \$150 of contribution per incremental sale, one of their models peaks at 40 percent volume with 680 incremental sales and about \$62,000 of profit, and another at 60 percent with 870 sales and about \$70,500. They also show a case where the total at full depth is lower than the total at 80 percent, because the last two deciles contain customers the treatment puts off — there, treating everyone loses incremental sales as well as money.

How precise the number is

A total incremental benefit is an estimate of a difference between two rates, multiplied by a population. It inherits the interval of that difference and magnifies it by the same factor. In the example the point estimate is 363.3 incremental conversions and the 95 percent interval runs from 217.9 to 508.7 — from about \$10,900 to about \$25,400 of value once the margin is applied.

That width is normal rather than a sign of a badly run campaign, and Radcliffe and Surry attach the same caution to their own profit figures: the calculation “assumes that the uplift estimates are accurate”, and “in general, uplift cannot be estimated as accurately as a purchase rate can be”. A total quoted to the nearest dollar, or two campaigns separated by a few percent, are both reading precision that is not there. Kohavi and colleagues’ interval for an absolute difference is the standard form, and it doubles as the test: if zero is inside it, the campaign has not been shown to work at all.

What to state alongside it

• The population it covers. A rate measured on an experiment’s participants, scaled to a larger audience that was never exposed, is an extrapolation — see incremental conversions.
• The cost. Both the per-contact cost and anything given away, such as the margin surrendered on a discount that a sure thing would have converted without.
• The interval. Carried through the margin multiplication, not dropped at the point where the number becomes a currency.
• The counterfactual. The total is measured against no treatment. Against the previous campaign rather than against nothing, it is a different and usually much smaller quantity.
• That randomization held. Without a randomized holdout, an unadjusted comparison does not give a counterfactual, and the figures above then measure the old targeting rule instead. An observational estimate needs justified identification assumptions and a suitable adjustment before it supports the same reading — see random targeting strategy.

References

• Nicholas J. Radcliffe and Patrick D. Surry, Real-World Uplift Modelling with Significance-Based Uplift Trees, Portrait Technical Report TR-2011-1, Stochastic Solutions White Paper, 2011
• Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne, “Controlled experiments on the web: survey and practical guide”, Data Mining and Knowledge Discovery 18(1), 2009