Incremental Revenue

Incremental revenue is the money a treatment caused, as opposed to the money that arrived while the treatment was running. Radcliffe and Surry give the continuous-outcome form directly: where a conventional model estimates \(E(R \mid T)\), the expected revenue among the treated, an uplift measurement estimates \(E(R \mid T) - E(R \mid C)\), the difference against a control group that was not treated.

With \(n_t\) treated customers generating total revenue \(R_t\) and \(n_c\) controls generating \(R_c\), over a population of \(N\),

\[ \text{incremental revenue} = \left( \frac{R_t}{n_t} - \frac{R_c}{n_c} \right) \cdot N \]

Revenue per customer is differenced, then scaled. Subtracting the two totals directly estimates the incremental revenue scaled to one arm’s size rather than to \(N\) — it is not the treated arm’s actual revenue, which includes everything those customers would have bought anyway — and with unequal arms it estimates neither; when the control group is twice the treated group it books roughly twice the revenue whatever the campaign did. Equal arms do not rescue the raw difference either, because the two forms answer to different \(N\). With 100 customers per arm booking \$2,000 and \$1,000 of revenue, the raw difference is \$1,000 and the scaled form with \(N = n_t + n_c = 200\) gives \$2,000. Both are estimates, and neither is a measured amount the campaign brought in: the control arm’s \$1,000 is not the treated arm’s own counterfactual, only a stand-in that randomization makes unbiased on average. In any single experiment the realized effect can differ — \$1,000 is perfectly consistent with a true effect of zero. So the \$1,000 estimates the effect scaled to one arm’s size, the \$2,000 estimates it for a population of 200, and \(N\) has to be fixed before either number means anything: one arm’s size, the tested population, or the rollout audience for a projection that additionally assumes the effect carries over.

Why revenue is harder to measure

A conversion is zero or one. Revenue is zero for most customers and occasionally very large, and that skew makes the same campaign far less precisely measured in money than in conversions. The data below are synthetic: 60,000 customers, 30 percent treated, a 5 percent base purchase rate rising by 1 percentage point under treatment, and a lognormal order value so that a few large orders dominate the total.

import numpy as np

def revenue_campaign(seed=11, N=60000, treated_share=0.3):
    """60,000 customers, 30 percent treated, where the outcome is revenue rather than a conversion.
    Buyers spend a lognormal amount, so a few orders dominate the total."""
    rng = np.random.default_rng(seed)
    treatment = (rng.random(N) < treated_share).astype(int)
    buys = rng.random(N) < 0.05 + treatment * 0.01
    spend = rng.lognormal(mean=4.0, sigma=1.2, size=N)
    return buys * spend, treatment

def difference_of_means(values, treatment):
    a, b = values[treatment == 1], values[treatment == 0]
    diff = a.mean() - b.mean()
    se = np.sqrt(a.var(ddof=1) / len(a) + b.var(ddof=1) / len(b))
    return diff, se

revenue, treatment = revenue_campaign()
N = len(revenue)
per_customer, se = difference_of_means(revenue, treatment)
print(f"{per_customer:.3f} {se:.3f} {per_customer * N:.0f} "
      f"{(per_customer - 1.96 * se) * N:.0f} {(per_customer + 1.96 * se) * N:.0f}")
# 0.324 0.443 19463 -32652 71578

converted = (revenue > 0).astype(float)
# A conversion rate on the same customers is far better determined.
rate, se_rate = difference_of_means(converted, treatment)
print(f"{rate * 100:.3f} {se_rate * 100:.3f} {rate / se_rate:.2f} {per_customer / se:.2f}")
# 0.641 0.203 3.16 0.73

margin, cost_per_contact = 0.30, 2.0
# Break-even: the incremental revenue per customer that just covers one contact.
print(f"{cost_per_contact / margin:.2f} {per_customer:.3f}")
# 6.67 0.324

The same experiment shows a conversion effect comfortably distinguishable from zero and a revenue effect that is not. Scaled to the population, the incremental revenue is \$19,463 with a 95 percent interval running from -\$32,652 to +\$71,578: the campaign might have made seventy thousand dollars or lost thirty. Nothing is wrong with the test; revenue simply carries the variance of the order-size distribution on top of the variance of whether anyone buys.

Radcliffe and Surry make the general version of the point about uplift measurement as a whole: “in general, uplift cannot be estimated as accurately as a purchase rate can be”. For revenue the gap is wider still. Three habits follow. Report conversions as well as revenue, since the conversion effect is what the data can actually resolve. Expect to need several times the traffic for a revenue readout at the same confidence. And treat trimming or winsorizing large orders as a decision to be declared, not a cleaning step, because it changes the estimand.

Revenue is not profit

Subtracting campaign cost from incremental revenue defines revenue net of that cost, but it does not establish profit: the goods and other incremental costs still need to be accounted for. The comparison the decision needs is between the contribution the incremental revenue carries and what the campaign spent to obtain it.

At a 30 percent contribution margin and \$2 to contact one customer, each contacted customer has to generate \$6.67 of incremental revenue to break even. The estimate above is \$0.32 per customer, twenty times short — so on these figures a blanket send loses money even though the campaign demonstrably lifts conversions. Targeting changes that arithmetic by concentrating the spend where the effect is, which is the subject of total incremental benefit, and the cost side is set out under treatment cost and revenue net of treatment cost.

Where incremental revenue goes missing

• Discounts to customers who would have bought anyway. Radcliffe and Surry describe the case: a customer who has decided to buy receives a coupon and uses it, so “not only were marketing costs increased by including the customer, but incremental revenue was reduced”. The attribution system records a success while the campaign has destroyed margin.
• Cannibalization. A promotion on one product can move demand from another, or pull forward purchases that would have happened next month. Measured on the promoted product alone the effect is positive; measured on the customer’s total spend over a longer window it can vanish.
• The window. Revenue within seven days and revenue within ninety are different variables, and a campaign that accelerates purchases looks strong on the first and flat on the second.

All three are reasons the estimand has to be stated with the number: which revenue, over what window, against what alternative. A figure that says only how much extra money arrived is not yet a measurement.

Related quantities

Each is the same causal comparison in a different unit, and each rests on the comparison described under random targeting strategy — a randomized holdout, or an adjusted estimate carrying the assumptions a holdout would have removed. Only the units, and the precision, change.

References

• Nicholas J. Radcliffe and Patrick D. Surry, Real-World Uplift Modelling with Significance-Based Uplift Trees, Portrait Technical Report TR-2011-1, Stochastic Solutions White Paper, 2011
• Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne, “Controlled experiments on the web: survey and practical guide”, Data Mining and Knowledge Discovery 18(1), 2009