A/B Testing and Experiment Design

An offline improvement may not translate into a better user experience. An A/B test assigns eligible units to a control version and a treatment version, then compares outcomes over a specified period. Random assignment makes the groups comparable in expectation; sampling variation, implementation errors, and interactions between units still affect what the experiment can establish.

Define the decision before collecting outcomes

Specify who is eligible, the versions being compared, the assignment unit, and a primary outcome over a fixed window. For example, compare seven-day purchases per assigned user under two ranking systems. Analyze users according to their assigned version (intention to treat), including those who never click; filtering on post-assignment engagement can create selection bias. Randomization supports a causal interpretation when outcomes are measured comparably and the design handles interference. It does not make each realized group identical.

The effect estimate is the treatment mean minus the control mean. A conversion rate rising from 10% to 11% is an absolute gain of 1 percentage point and a relative gain of 10%. Report the estimate and a confidence interval, then compare them with a practically worthwhile effect. An interval crossing zero does not establish no effect; excluding zero does not establish a worthwhile gain. A guardrail is an outcome whose unacceptable deterioration could prevent rollout even when the primary metric improves.

For an illustrative fixed-horizon comparison of independent user outcomes, suppose the treatment and control means are 1.04 and 1.00, each arm has 10,000 users, and both sample standard deviations are 1. The estimated standard error is \(\sqrt{1^2/10{,}000+1^2/10{,}000}\approx0.0141\). A large-sample 95% interval is \(0.04\pm1.96\sqrt{2/10{,}000}\), approximately [0.0123, 0.0677]. It excludes zero but still includes effects below a worthwhile gain of 0.02. This calculation assumes independent units and a suitable normal approximation; clustered requests require the adjustment below.

Choose a minimum detectable effect (MDE), a false-positive budget such as 5%, and a desired power such as 80%: the chance of rejecting the null at that specified effect under the planning model. Exercise 1 computes the implied sample size. Also allow the outcome window to mature and cover relevant calendar cycles. Sample size alone does not resolve delayed outcomes or changing user behavior.

Keep assignment stable and log assignment separately from exposure and outcomes. Check whether arm counts match the planned allocation (sample ratio mismatch), investigate missing or duplicated events, and check whether filtering differs by arm. A/A tests, which serve identical versions through the experimental pipeline, can expose some calibration and logging failures. Passing those checks does not prove that the experiment is valid.

Account for the randomization unit

Suppose a ranking change is assigned once per user, but each user generates several requests. A user’s requests share preferences and circumstances. Treating all requests as independent can understate uncertainty even though the assignment itself was properly randomized. In this simulation, rows of y are users and columns are requests; arm selects the treatment users. Each request adds independent noise to a shared user component, and the true treatment effect is zero.

import numpy as np
from scipy import stats

n_users, reqs = 4000, 12
rej_request, rej_user = [], []
for rep in range(400):
    g = np.random.default_rng(rep)
    arm = g.random(n_users) < 0.5
    user_level = g.normal(0, 1.0, n_users)                  # per-user propensity
    y = user_level[:, None] + g.normal(0, 1.0, (n_users, reqs))   # TRUE effect is zero
    a, b = y[arm].ravel(), y[~arm].ravel()
    rej_request.append(stats.ttest_ind(a, b, equal_var=False).pvalue < 0.05)
    rej_user.append(stats.ttest_ind(y[arm].mean(1), y[~arm].mean(1),
                                    equal_var=False).pvalue < 0.05)

print(f"request-level t-test rejects {np.mean(rej_request):.4f}   (nominal 0.05)")
print(f"user-level    t-test rejects {np.mean(rej_user):.4f}")
icc = 1.0 / (1.0 + 1.0)
print(f"intra-user correlation {icc:.2f}; design effect 1+(m-1)*icc = {1 + (reqs - 1) * icc:.1f}")
# request-level t-test rejects 0.4400   (nominal 0.05)
# user-level    t-test rejects 0.0475
# intra-user correlation 0.50; design effect 1+(m-1)*icc = 6.5

Across these 400 null experiments, the request-level test rejects 44% of the time and the user-level test rejects 4.75%. These are simulation estimates, not exact error probabilities. At a true rate of 5%, the Monte Carlo standard error from 400 repetitions is about 1.1 percentage points.

For independent clusters of equal size \(m\), equal marginal request variance, and a common within-user correlation \(\rho\), the mean’s variance is inflated relative to independent requests by the design effect \(D=1+(m-1)\rho\). Here \(\rho=1/(1+1)=0.5\) and \(D=6.5\). For estimating this mean, 48,000 requests have the variance-equivalent sample size \(48{,}000/6.5\approx7{,}385\) independent requests. Ignoring clustering understates the standard error by approximately \(\sqrt{6.5}\approx2.55\). This formula is not a universal effective sample size for unequal clusters or other statistics.

For a user-average outcome, aggregate within each user and compare independent user summaries. A request-level analysis can instead use user-clustered standard errors, with enough independent users and an appropriate model. If request counts differ, an average of user means and a pooled request mean weight users differently and target different quantities. Ratio metrics need uncertainty calculations that account for their numerator, denominator, and clustering. More requests per user can help: here the variance of a user mean is \(1+1/m\), falling from 2 at one request to about 1.083 at twelve, then toward 1. Additional requests reduce request noise but do not remove the shared user component.

Plan interim decisions

K, n_final = 10, 20000

def looks(rep, effect):
    g = np.random.default_rng(rep)
    a = g.normal(effect, 1, n_final); b = g.normal(0, 1, n_final)
    sizes = np.arange(1, K + 1) * (n_final // K)
    def summaries(x):
        sums = np.cumsum(x)[sizes - 1]
        squares = np.cumsum(x * x)[sizes - 1]
        mean = sums / sizes
        sd = np.sqrt((squares - sums * sums / sizes) / (sizes - 1))
        return mean, sd
    ma, sa = summaries(a)
    mb, sb = summaries(b)
    return stats.ttest_ind_from_stats(ma, sa, sizes, mb, sb, sizes,
                                     equal_var=False).pvalue

for label, eff in (("no true effect", 0.0), ("true effect 0.04", 0.04)):
    naive = bonf = final = 0
    for rep in range(2000):
        p = looks(rep, eff)
        naive += (p < 0.05).any()          # stop the moment it looks significant
        bonf  += (p < 0.05 / K).any()      # the same looks, corrected
        final += (p[-1] < 0.05)            # no peeking, one test at the end
    print(f"{label:17s} peek-and-stop {naive/2000:.4f}   Bonferroni over looks {bonf/2000:.4f}"
          f"   single final test {final/2000:.4f}")
# no true effect    peek-and-stop 0.2065   Bonferroni over looks 0.0290   single final test 0.0595
# true effect 0.04  peek-and-stop 0.9825   Bonferroni over looks 0.8965   single final test 0.9715

Here each arm reaches 20,000 independent observations, with ten equally spaced cumulative looks. Reusing a 5% threshold at every look yields a 20.65% estimated false-positive rate under the null. The looks overlap and are dependent, so this rate cannot be computed by pretending there are ten independent tests. Simply viewing data is not the error; using unadjusted significance repeatedly to make the decision is.

Bonferroni uses 0.05/10 at each of these ten planned looks. The union bound controls the probability of any rejection under the null at no more than 5% when the individual tests are valid, regardless of their dependence. Its observed rate is 2.90%; the single-final-test estimate is 5.95%. With 2,000 repetitions, Monte Carlo uncertainty near 5% has a standard error of about 0.49 percentage points. The true-effect row compares detection rates: 89.65% for this correction and 97.15% for the final test. The naive 98.25% is a real rejection frequency, but it is not a fair power comparison at the same false-positive rate.

If early decisions are unnecessary, predefine the sample size, measurement window, and analysis time. If they are necessary, choose a sequential method in advance. Group-sequential boundaries can allocate the total error probability across planned looks; O’Brien–Fleming-style boundaries are stringent early and closer to the fixed-sample threshold late. A confidence sequence covers the target simultaneously across observation times under its stated assumptions, allowing data-dependent stopping. These methods are not interchangeable with ordinary confidence intervals recalculated every day. Safety monitoring may justify stopping regardless of significance, but the subsequent analysis must reflect that stopping rule.

CUPED: using pre-experiment measurements

A pre-experiment user metric can explain some variation in the outcome. CUPED (Controlled-experiment Using Pre-Experiment Data) uses that relationship to improve precision. It is most useful when the pre-period covariate predicts the experiment-period outcome; new users, missing history, or weak correlation can limit the gain.

Let X be the pre-period metric and Y the experiment outcome. Write the adjusted outcome as \(Y_c=Y-\theta(X-\bar X)\). For a fixed coefficient and a covariate independent of randomized assignment, the treatment–control mean difference subtracts \(\theta(\bar X_T-\bar X_C)\), whose expectation is zero. The variance-minimizing population coefficient is \(\operatorname{Cov}(Y,X)/\operatorname{Var}(X)\). The code estimates it from the same experiment, so that exact fixed-coefficient argument does not automatically give finite-sample unbiasedness; standard regression-adjustment inference requires its own conditions. Both covariance and variance in the code use the same normalization.

n = 20000
plain, cuped = [], []
for rep in range(300):
    g = np.random.default_rng(rep)
    pre = g.normal(0, 1, n)                       # the same metric, before the experiment
    arm = g.random(n) < 0.5
    y = 0.8 * pre + g.normal(0, 0.6, n) + 0.02 * arm      # true lift 0.02
    plain.append(y[arm].mean() - y[~arm].mean())
    theta = np.cov(y, pre, ddof=0)[0, 1] / np.var(pre)
    yc = y - theta * (pre - pre.mean())           # CUPED
    cuped.append(yc[arm].mean() - yc[~arm].mean())

print(f"plain  mean {np.mean(plain):+.5f}  sd {np.std(plain, ddof=1):.5f}")
print(f"CUPED  mean {np.mean(cuped):+.5f}  sd {np.std(cuped, ddof=1):.5f}")
print(f"variance reduction {1 - np.var(cuped, ddof=1) / np.var(plain, ddof=1):.4f}"
      f"   sample-size multiplier {np.var(plain, ddof=1) / np.var(cuped, ddof=1):.2f}x")
# plain  mean +0.02055  sd 0.01415
# CUPED  mean +0.02044  sd 0.00855
# variance reduction 0.6350   sample-size multiplier 2.74x

The repeated estimates average 0.02055 and 0.02044, close to the simulated effect 0.02. Their standard deviations are about 0.01415 and 0.00855. The observed variance ratio corresponds to a 2.74-fold sample-size multiplier if variance continues to scale inversely with independent user count. It is a comparison within this setup, not free traffic or a guaranteed reduction in calendar duration. Close simulation averages alone do not prove unbiasedness.

Use covariates whose values cannot be affected by treatment, and plan how missing history is handled. Measurement during the experiment does not by itself make a covariate a mediator: the issue is whether assignment can affect it or its selection. Adjusting for a treatment-affected variable can bias the comparison or change the target. A near-constant pre-period covariate also needs handling because its estimated variance is near zero. For inference, use a suitable regression-adjustment method or design-based variance estimate; the simulation reports variation across whole experiments rather than a confidence interval from one experiment.

Interference belongs in the design

If treated buyers take inventory that control buyers would otherwise have purchased, the control outcome also depends on treatment allocation. The difference between arms then measures an effect in that mixed marketplace. It need not equal the change when everyone receives the new ranking. Exercise 2 separates the arm contrast from the overall change. Isolating markets or randomizing system-level time blocks can help, but the design must account for spillovers and carryover.

Exercises

1. How many users does the claim need? Compute a normal-approximation sample size for an absolute mean difference. Distinguish independent requests, correlated requests, and users contributing twelve requests each. Here request standard deviation is 1, unlike the square-root-of-two value in the first simulation.

Track the units in every column and round the required user count upward.

import numpy as np
from scipy import stats
sd, alpha, power = 1.0, 0.05, 0.80
z = stats.norm.ppf(1 - alpha / 2) + stats.norm.ppf(power)
reqs, icc = 12, 0.5
design = 1 + (reqs - 1) * icc
print("absolute MDE | iid requests/arm | users/arm | correlated requests/arm")
for mde in (0.20, 0.10, 0.05, 0.02, 0.01):
    n_iid = z ** 2 * 2 * sd ** 2 / mde ** 2
    users = int(np.ceil(n_iid * design / reqs))
    print(f"{mde:12.2f} | {int(np.ceil(n_iid)):16,} | {users:9,} | {users * reqs:23,}")
# absolute MDE | iid requests/arm | users/arm | correlated requests/arm
#         0.20 |              393 |       213 |                   2,556
#         0.10 |            1,570 |       851 |                  10,212
#         0.05 |            6,280 |     3,402 |                  40,824
#         0.02 |           39,245 |    21,258 |                 255,096
#         0.01 |          156,978 |    85,030 |               1,020,360

2. Interference changes the comparison. Let treatment raise treated outcomes while lowering control outcomes in the same experiment. Compare the arm difference with the population-average change at 50% treatment saturation.

The arm difference and the average change over all users measure different quantities.

import numpy as np
n, gain = 20000, 0.10
for spill in (0.0, 0.25, 0.5, 1.0):
    contrasts, population_changes = [], []
    for rep in range(200):
        g = np.random.default_rng(rep)
        arm = g.permutation(n) < n // 2
        base = g.normal(1.0, 1.0, n)
        y = base + arm * gain - (~arm) * spill * gain
        contrasts.append(y[arm].mean() - y[~arm].mean())
        population_changes.append((y - base).mean())
    print(f"spill {spill:.2f}  arm contrast {np.mean(contrasts):+.4f}"
          f"  population change {np.mean(population_changes):+.4f}")
# spill 0.00  arm contrast +0.0996  population change +0.0500
# spill 0.25  arm contrast +0.1246  population change +0.0375
# spill 0.50  arm contrast +0.1496  population change +0.0250
# spill 1.00  arm contrast +0.1996  population change +0.0000

3. A dashboard of guardrail metrics. Simulate 1, 5, 20, and 50 independent null p-values. Count how often any test rejects, and distinguish family-wise error control from false discovery rate control.

With twenty independent null tests, the uncorrected alarm probability is about 64%; with fifty, about 92%.

import numpy as np

for n_metrics in (1, 5, 20, 50):
    flag, bonf, fdr = [], [], []
    for rep in range(2000):
        g = np.random.default_rng(rep)
        p = g.random(n_metrics)                    # every metric is null: p is uniform
        flag.append((p < 0.05).any())
        bonf.append((p < 0.05 / n_metrics).any())
        srt = np.sort(p)
        fdr.append((srt <= 0.05 * np.arange(1, n_metrics + 1) / n_metrics).any())
    print(f"{n_metrics:3d} metrics, all null:  any p<0.05 {np.mean(flag):.4f}"
          f"   Bonferroni {np.mean(bonf):.4f}   Benjamini-Hochberg {np.mean(fdr):.4f}")
# 1 metrics, all null:  any p<0.05 0.0540   Bonferroni 0.0540   Benjamini-Hochberg 0.0540
#   5 metrics, all null:  any p<0.05 0.2430   Bonferroni 0.0535   Benjamini-Hochberg 0.0550
#  20 metrics, all null:  any p<0.05 0.6535   Bonferroni 0.0485   Benjamini-Hochberg 0.0490
#  50 metrics, all null:  any p<0.05 0.9275   Bonferroni 0.0510   Benjamini-Hochberg 0.0525

References

Deng et al., CUPED; Howard et al., confidence sequences; Bojinov et al., switchback experiments; Microsoft Research, diagnosing sample ratio mismatch. The distinctions between estimation, uncertainty, and repeated testing are developed in Statistical Inference for Machine Learning.