Conversion Rate (CR)

A conversion rate is the share of some population that took a desired action. Kohavi and colleagues give the e-commerce version precisely: it is “the percentage of visits to the website that include a purchase”. What counts as a conversion depends on the business — a purchase, a sign-up, a form submission, a subscription — and what counts in the denominator decides almost everything else about the number.

\[ \text{CR} = \frac{\text{conversions}}{\text{units exposed}} \]

With 10,000 visits of which 500 include a purchase, the conversion rate is 5 percent per visit. The count that matters is converting visits: if one visit can contain several purchases, the number of purchases and the number of visits that include a purchase differ, and only the latter gives the share of visits that include a purchase. Stated that way the figure is complete; stated as “5 percent” it is not, and the next section is why.

The denominator is the metric

These are four different numbers on the same data, and they can move in opposite directions at the same time. In a hypothetical redesign, the user-based rate can rise while the visit-based rate falls if more users buy but visits per user grow faster. A campaign aimed at narrower traffic can raise the click-based rate while reducing total conversions if it brings fewer clicks overall. Quoting one as though it were the conversion rate, or comparing this quarter’s user-based rate with last quarter’s visit-based rate, produces confident nonsense.

Where the denominator can be gamed

Because the metric is a ratio, it rises when the denominator shrinks while the numerator holds — and the denominator is a definition someone maintains. Excluding single-page visits, filtering out traffic from a low-intent source, or tightening what counts as a lead can raise the conversion rate without changing how many people bought anything, when the excluded units contained few or no conversions.

Some of those exclusions are legitimate. Kohavi and colleagues devote a section to robot traffic precisely because automated requests distort experimental results, and removing them makes the metric more meaningful rather than less. The test is whether the exclusion was defined before the number was needed and applied to every period being compared. A filter introduced in the same quarter as the improvement it produces needs that scrutiny. A later correction can be legitimate when it is disclosed and the periods being compared are recomputed under the same definition.

The defense is to report the numerator and denominator alongside the ratio. A conversion rate that rose from 4.8 to 5.4 percent while conversions fell is visible immediately in the counts and invisible in the percentage.

When the denominator is not the unit of analysis

This is the failure that survives peer review inside a company, because the resulting number looks right and its error bars do not.

Experiments normally randomize by user: a person is assigned to a variant and stays there. If the metric is a rate per visit, the analysis treats each visit as an independent observation, and one user’s five visits are not five independent draws — they share that user’s preferences, device and intent. Treating them as independent understates the standard error, so intervals come out too narrow and tests reject too often. The effect is larger the more visits per user the product generates.

The remedy is to analyze at the unit that was randomized — aggregate each user to a single observation — or to use a standard error that accounts for the clustering. Switching to a different test does not help, because the problem is in the data’s structure rather than in the test. Two-proportion z-test assumes independent observations for exactly this reason.

The number on the dashboard is an estimate

A displayed conversion rate is a sample quantity standing in for an unobservable one, and it carries a width even when the dashboard shows none. On 1,000 visits at an observed 5 percent, the plausible range for the underlying rate spans roughly 3.8 to 6.5 percent — wide enough that most week-to-week movement in a small funnel step is sampling variation with a narrative attached. True conversion rate covers the parameter and the fact that the textbook interval around it covers less often than it claims, with coverage falling to 87 percent for a 1 percent rate on 500 observations.

Comparing two rates is a further step again: whether the difference is real is the subject of conversion rate uplift, and whether the difference was caused by the change requires the randomization described under A/B testing, or an adjustment that stands in for it. Without either, an improvement measured before and after is confounded with everything else that moved.

What it connects to

Conversion rate is the hinge between engagement metrics and money. Click-through rate measures whether people engage with an advertisement; conversion rate measures whether that engagement produces the outcome. The acquisition cost follows arithmetically: when the two are measured over the same denominator, cost per acquisition is cost per click divided by the conversion rate, so doubling the rate halves the cost of a customer without any change in what traffic costs.

That relationship is what makes conversion rate optimization worth doing — clearer calls to action, faster pages, fewer form fields, less checkout friction, visible trust signals — and it is also why the temptation to improve the metric by redefining it is persistent. One further caution: the rate counts conversions that happened, not conversions the change caused. Some of the purchases in the numerator would have happened anyway, which is the gap between a conversion rate and incremental conversions.

References

• Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne, “Controlled experiments on the web: survey and practical guide”, Data Mining and Knowledge Discovery 18(1), 2009
• Lawrence D. Brown, T. Tony Cai and Anirban DasGupta, “Interval Estimation for a Binomial Proportion”, Statistical Science 16(2), 101–133, 2001