Incremental Conversions

Incremental conversions are the conversions that happened only because of a campaign. Total conversions count everyone who converted; incremental conversions count the ones that would not have converted otherwise. The gap between the two is the whole subject, and it is usually large: most people who buy after seeing an advertisement were going to buy anyway.

How they are computed

With a randomized holdout, \(n^T\) treated users producing \(y^T\) conversions and \(n^C\) control users producing \(y^C\),

\[ \text{incremental conversions} = \left( \frac{y^T}{n^T} - \frac{y^C}{n^C} \right) \cdot N \]

Conversion rates are differenced, and the result is scaled by \(N\). The shorter form, treated conversions minus control conversions, answers a different question: it estimates the effect scaled to one arm’s size, and matches the scaled form only when \(N\) is that size and the arms are equal. In most real campaigns the holdout is deliberately small, so it is not even a clean arm-level figure. When the control arm is twice the treated arm it produces roughly twice the conversions, so the raw difference can be negative while the campaign is working well; incremental gain shows a case where that happens in nine deciles out of ten.

Equal arms do not rescue the raw difference either, because the two forms answer to different \(N\). With 100 customers per arm converting 20 and 10, the raw difference is 10 and the scaled form with \(N = n_t + n_c = 200\) gives 20. Both are estimates, and neither is a count of what the campaign caused: the control arm’s 10 conversions are not the treated arm’s own counterfactual, only a stand-in for it that randomization makes unbiased on average. In any single experiment the realized effect on the treated can differ — 10 is perfectly consistent with a true effect of zero.

So the 10 estimates the effect scaled to one arm’s size, the 20 estimates it for a population of 200, and \(N\) has to be fixed before either number means anything: one arm’s size, the tested population, or the rollout audience for a projection that additionally assumes the effect carries over.

The unit being counted is part of the definition. Kohavi and colleagues define the underlying metric as “the percentage of visits to the website that include a purchase”, and a rate per visit is a different number from a rate per user on the same data. A campaign that makes people return more often moves the two in different directions.

What goes in N

The scaling factor decides what the number means, and it is the part most often left unstated.

The third row is a projection wearing the clothes of a measurement. It can be reasonable — if the test population was drawn at random from the audience in question, it is close to the first row — and it is often not, because tests run on active users and roll-outs reach lapsed ones. Say which row a figure belongs to, and a forecast stops being mistaken for a result.

Incremental is not attributed

Most conversion reporting counts conversions that can be tied to the campaign: a purchase within a fixed window after the email, a coupon code, a click-through. Radcliffe and Surry explain why that is a different quantity. Such rules catch purchases that would have happened regardless while missing incremental purchases from people who were influenced but never used the code.

Their coupon case is the clearest. A customer who has already decided to buy receives a discount coupon and uses it — as helpful staff will often encourage. The attribution system records a success, and “manifestly, in these cases, the sales are not incremental”; the campaign has both spent money on that customer and reduced the revenue it would have collected anyway. Attributed conversions can therefore exceed incremental conversions by a wide margin, and no amount of care in the attribution rule fixes it, because the rule has no access to what would have happened.

How precise the count is

Incremental conversions are a difference of two observed rates multiplied by a population, so the interval on that difference is magnified by the same factor. On the synthetic campaign used under total incremental benefit — 20,000 customers, 30 percent treated, a 5 percent base rate — the effect is 1.82 percentage points and the point estimate is 363.3 incremental conversions, with a 95 percent interval from 217.9 to 508.7.

A range that wide on a campaign of 20,000 is normal rather than a sign of a badly run test, and it is why two campaigns whose incremental conversions differ by ten percent have usually not been distinguished. Kohavi and colleagues note that the interval also serves as the test: when zero falls inside it, the campaign has not been shown to have worked. Confidence intervals and two-proportion z-test cover the arithmetic; sample size covers how much traffic a given precision needs.

Where it leads

Incremental conversions are the count form of the campaign’s effect. Expressed as a rate against the baseline they become conversion rate uplift; accumulated down a ranking they become cumulative incremental gain; multiplied by a margin and netted against the spend they become the profit calculation under total incremental benefit. The raw randomized comparison shown here requires a holdout, as random targeting strategy explains. Observational estimates instead require an explicit identification argument and suitable adjustment; attribution alone supplies neither.

References

• Nicholas J. Radcliffe and Patrick D. Surry, Real-World Uplift Modelling with Significance-Based Uplift Trees, Portrait Technical Report TR-2011-1, Stochastic Solutions White Paper, 2011
• Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne, “Controlled experiments on the web: survey and practical guide”, Data Mining and Knowledge Discovery 18(1), 2009