True Conversion Rate

The true conversion rate is the parameter an experiment is trying to estimate: the probability that a unit drawn from a defined population converts under defined conditions. It is fixed and unobservable. What a dashboard displays is the sample conversion rate \(\hat{p} = x/n\), a random quantity that lands near the true rate and rarely on it.

Keeping the two apart is the whole point of the term. Two weeks of a landing page at 5.0 percent and the next two at 5.3 percent may be two draws from the same process; whether it is evidence of a change depends on how many visits are behind each figure, which the percentages alone do not say. That is why the same distinction underlies two-proportion z-test and confidence intervals.

What the parameter is attached to

A true conversion rate is not a universal constant belonging to a page or a product. It is defined relative to a population, a period, and a set of conditions, and it moves when any of them move. Describing it as the rate among everyone who could ever see version A is not a definition, because that group is not enumerable and its composition changes with every campaign that sends traffic.

The denominator matters just as much. Kohavi and colleagues define the metric in terms of visits — “the percentage of visits to the website that include a purchase” — and a rate per visit and a rate per user are different parameters over the same people. A change that makes visitors return more often before buying lowers the first and leaves the second alone. Neither is more correct; an estimate of one is not an estimate of the other, and the choice belongs next to the number. The metric itself is covered under conversion rate.

The interval, and what it claims

The textbook interval is \(\hat{p} \pm z_{\alpha/2}\sqrt{\hat{p}(1 - \hat{p})/n}\). With 50 conversions from 1,000 visitors it gives about 3.65 to 6.35 percent around an observed 5 percent.

What that interval says is precise and easy to overstate. The true rate is fixed; the interval is the random object. The 95 percent describes the procedure — intervals built this way from repeated samples contain the true rate about 95 percent of the time — and not the probability that this particular interval contains it. The distinction sounds pedantic until someone treats the interval as a distribution over the parameter, which is a different calculation and the subject of posterior probability of uplift.

About 95 percent, except when it is not

The stronger problem is that this interval does not deliver the coverage it advertises. Brown, Cai and DasGupta revisited it and were blunt: “the chaotic coverage properties of the Wald interval are far more persistent than is appreciated”, and “common textbook prescriptions regarding its safety are misleading and defective in several respects and cannot be trusted”.

Coverage can be computed exactly rather than simulated, by enumerating every possible number of conversions and weighting each by its binomial probability.

import numpy as np
from scipy import stats

def wald(x, n, z=1.96):
    p = x / n
    half = z * np.sqrt(p * (1 - p) / n)
    return p - half, p + half

def wilson(x, n, z=1.96):
    p = x / n
    center = (p + z**2 / (2 * n)) / (1 + z**2 / n)
    half = z / (1 + z**2 / n) * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2))
    return center - half, center + half

def coverage(p, n, interval):
    """Exact coverage: every possible count, weighted by its binomial probability."""
    counts = np.arange(n + 1)
    low, high = interval(counts, n)
    covered = (low <= p) & (p <= high)
    return stats.binom.pmf(counts, n, p)[covered].sum()

for p, n in ((0.05, 1000), (0.05, 200), (0.01, 500), (0.30, 100)):
    print(f"{p:.2f} {n:>4} {coverage(p, n, wald):.4f} {coverage(p, n, wilson):.4f}")
# 0.05 1000 0.9417 0.9504
# 0.05  200 0.9256 0.9672
# 0.01  500 0.8714 0.9623
# 0.30  100 0.9502 0.9372

low, high = wald(50, 1000)
lo_w, hi_w = wilson(50, 1000)
print(f"{low:.4f} {high:.4f} {lo_w:.4f} {hi_w:.4f}")
# 0.0365 0.0635 0.0381 0.0653

The third row is the one to remember: an interval labeled 95 percent that contains the true rate 87 percent of the time, on a sample of 500 — a perfectly ordinary size for a low-converting funnel step. The first row shows the shortfall is already present at a thousand observations. The last row is included because it cuts the other way: Wald is not uniformly worse, and Wilson’s coverage can sit slightly below nominal too. What distinguishes them is that Wilson stays close to the target across the range while Wald degrades badly wherever the rate is small.

Brown, Cai and DasGupta recommend the Wilson interval or an equal-tailed Jeffreys interval for small \(n\), and the Agresti–Coull interval for larger \(n\). For the 50-from-1,000 example, Wilson gives 3.81 to 6.53 percent against Wald’s 3.65 to 6.35 — shifted upward and asymmetric, which is the right shape for a proportion bounded below by zero.

Why it matters in an experiment

An A/B test never observes either true rate. It observes two sample rates and asks whether their difference is larger than sampling variation would comfortably produce — see conversion rate uplift for the difference and its interval, and sample size for how much traffic a given precision requires.

Two habits follow from the distinction. Treat a headline rate as an estimate with a width, not a fact, especially when it comes from a segment small enough to be interesting. And before explaining a movement, check it against that width rather than against intuition — a difference inside the interval is not established, which is not the same as being established to be noise. A weekly conversion rate wandering between 4.8 and 5.3 percent on a few thousand visits is consistent with a constant true rate being sampled — which is a reason not to build a narrative on each week, not a demonstration that nothing moved. A real drift of that size would look much the same, and separating the two takes more data or a test designed for a trend.

References

• Lawrence D. Brown, T. Tony Cai and Anirban DasGupta, “Interval Estimation for a Binomial Proportion”, Statistical Science 16(2), 101–133, 2001
• Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne, “Controlled experiments on the web: survey and practical guide”, Data Mining and Knowledge Discovery 18(1), 2009