Incremental Sales

Incremental sales are the units a campaign caused — the difference between what was sold with the treatment and what would have been sold without it. The phrase is the business-side name for the quantity that uplift work calls an effect, and Radcliffe and Surry label the vertical axis of their evaluation charts exactly this way: an estimate of “the number of incremental sales achieved”.

With \(n_t\) treated customers buying \(u_t\) units and \(n_c\) controls buying \(u_c\), over a population of \(N\),

\[ \text{incremental sales} = \left( \frac{u_t}{n_t} - \frac{u_c}{n_c} \right) \cdot N \]

Units per customer are differenced, then scaled. Subtracting the two totals estimates the incremental units scaled to one arm’s size rather than to \(N\) — not the treated arm’s total units, which include the baseline — and with unequal arms it estimates neither; a holdout that is a third of the population books about half as many units as the treated arm whatever the campaign achieved. Equal arms do not rescue the raw difference either, because the two forms answer to different \(N\). With 100 customers per arm buying 200 and 100 units, the raw difference is 100 units and the scaled form with \(N = n_t + n_c = 200\) gives 200. Both are estimates, and neither is a count of units the campaign caused: the control arm’s 100 units are not the treated arm’s own counterfactual, only a stand-in that randomization makes unbiased on average. In any single experiment the realized effect can differ — 100 is perfectly consistent with a true effect of zero. So the 100 estimates the effect scaled to one arm’s size, the 200 estimates it for a population of 200, and \(N\) has to be fixed before either number means anything: one arm’s size, the tested population, or the rollout audience for a projection that additionally assumes the effect carries over.

Units, money, and profit

The same comparison produces different numbers depending on what is counted, and the distinction is not cosmetic. Suppose an advertisement moves a treated group from 1,000 units to 1,200 on equal-sized arms — figures chosen for illustration.

Only the last row is a decision. Multiplying 200 units by a \$50 margin gives \$10,000 of incremental contribution, which is not \$10,000 of incremental sales value and not \$10,000 of profit — the campaign still has to be paid for, which is the subject of treatment cost and revenue net of treatment cost.

When the units and the money disagree

A discount promotion is the standard case where the two diverge. It reliably lifts units, and it lowers the price on every unit sold — including the ones that would have sold anyway. Radcliffe and Surry describe the mechanism for a customer who had already decided to buy and then uses the coupon: “not only were marketing costs increased by including the customer, but incremental revenue was reduced”. A campaign can therefore show a clear positive effect on incremental sales and a negative effect on incremental revenue, and both measurements are correct.

Two related cases behave the same way. A promotion that shifts demand between products raises incremental sales of the promoted item while leaving the customer’s total basket unchanged. And one that pulls purchases forward lifts the seven-day window and flattens the ninety-day one. In each case the fix is not a better estimator but a better-specified outcome: which units, over what window, at what price.

How precisely it can be known

Units per customer is an average over a distribution that is mostly zeros with an occasional bulk order, so the estimate is noisier than a conversion rate on the same customers. Radcliffe and Surry state the general version: “in general, uplift cannot be estimated as accurately as a purchase rate can be.” In the worked comparison under incremental revenue, a campaign whose conversion effect is clearly distinguishable from zero has a money effect whose interval spans both a large gain and a large loss.

One further caution applies when sales are reported by segment. Raw segment estimates are not additive when the treated share differs between segments: Radcliffe and Surry give a case where two segments at 0.711 and 0.284 percentage points average to 0.497 while the combined population gives 0.500, and note that “this is not the result of rounding error”. So a decile table built from raw segment estimates will not sum to the campaign total, and the discrepancy is a property of the arithmetic rather than a reconciliation error to be hunted down. A decomposition that does add up defines every segment with the same fixed population weights.

Incremental sales and incremental gain

The two terms describe the same comparison from different sides of the business. Incremental sales is the reporting term for the whole campaign; incremental gain is the analytics term for what one slice of a ranking contributes, and cumulative incremental gain builds up over those slices by comparing treated and control directly on each selected prefix, not by summing separate slice estimates. Radcliffe and Surry move between them freely, relabeling the same chart axis from sales volume to sales value as the question changes.

What does not change is the requirement underneath. Every version of the number is a comparison against customers who were not treated, and it measures the campaign directly only if those customers were chosen at random; otherwise it needs adjustment and the assumptions that come with it — see random targeting strategy.

References

• Nicholas J. Radcliffe and Patrick D. Surry, Real-World Uplift Modelling with Significance-Based Uplift Trees, Portrait Technical Report TR-2011-1, Stochastic Solutions White Paper, 2011
• Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne, “Controlled experiments on the web: survey and practical guide”, Data Mining and Knowledge Discovery 18(1), 2009