Traditional A/B Test (Fixed-Horizon A/B Test)

A traditional A/B test is the fixed-horizon design applied to a product: split users into two arms, follow a prespecified sample-size or time-based stopping rule, and perform the planned confirmatory comparison. The statistical half of that — the sample size arithmetic, the stopping rule, what peeking costs — is fixed-horizon testing, which explains how the planned horizon relates to the test’s error control.

Random assignment supports a causal comparison when assignment, observation and analysis preserve the intended design. Dependence among observations must be reflected in uncertainty estimates. An effect need not be constant over time for a valid finite-period estimand, but that estimand may differ from a long-run effect.

Before assignment, define who is eligible, what A and B deliver, the randomization unit, and the outcome window. In a user-randomized checkout experiment, keep each user in the assigned variant and define conversion consistently in both arms. Comparing assigned users estimates the effect of offering the variant under that delivery process. Restricting analysis afterward to users who completed checkout can select a treatment-affected population and answer a different question.

For example, if independent user-level observations yield 500 conversions among 10,000 controls and 550 among 10,000 treated users, the observed difference is 0.5 percentage points, or 10% relative to the 5% control rate. An approximate unpooled standard error of the absolute difference is √[0.05 × 0.95/10,000 + 0.055 × 0.945/10,000] ≈ 0.00315. A 95% normal interval is about −0.12 to +1.12 percentage points: the positive estimate alone does not settle whether the variant improves conversion. This approximation assumes independent Bernoulli outcomes and adequate counts.

Choose the primary metric, uncertainty procedure, and decision criteria before reading results, and account for additional comparisons where they affect the error claim. A statistically detectable improvement may still be too small to justify implementation cost or a deterioration in another important outcome. The assignment and logging checks below establish whether the comparison can be trusted in the first place.

The split itself can be wrong

Fabijan and colleagues describe sample ratio mismatch as a symptom of several data-quality problems and emphasize diagnosing its cause. A mismatch is an investigation trigger; its consequences depend on whether assignment, exposure or observation was distorted.

Suppose a null-effect synthetic experiment assigns 100,186 users to control and 99,814 to treatment. A treatment-side logging fault preferentially loses lower-activity users, leaving 95,357 observed treated users. The observed treatment share is 95,357/(100,186 + 95,357) ≈ 48.77%. Because activity also predicts conversion, selection changes the observed outcome rate as well as the arm count. Even perfect arithmetic on the remaining records cannot reconstruct the missing comparison without addressing that mechanism.

The fault can make treatment appear beneficial even though it changes no customer behavior. Compare counts at assignment, exposure, and outcome collection to locate where the imbalance appears. An expected allocation is a probability, not a requirement for exactly equal counts, so assess deviations against the randomization design rather than treating any difference as a fault.

In this constructed example, the loss occurs after assignment and causes selection bias in the observed sample. A ratio check on observed records diagnoses a different stage from a ratio check on the original assignment log. A significant mismatch in another experiment requires root-cause investigation rather than an automatic claim that all possible analyses are invalid.

The unit you randomize is not the unit you count

User-level assignment can support a session-level metric. The error is treating sessions from the same user as independent observations. Use an analysis aligned with the estimand and a variance estimator or resampling scheme that accounts for user clusters.

The NumPy/SciPy experiment below creates a shared random effect for each user plus independent session noise. The parameter rho controls the within-user correlation. It repeats a null-effect experiment 1,500 times, comparing a Welch test that incorrectly pools sessions as independent with a test of each user’s mean. Equal session counts make the user-mean comparison appropriate for this illustration; unequal counts or treatment-dependent session counts require choosing the estimand and analysis deliberately.

import numpy as np
from scipy.stats import ttest_ind

rng = np.random.default_rng(555)
users, sessions = 4000, 12
reps = 1500


def one_trial(rho):
    arm = rng.random(users) < 0.5
    user_effect = rng.normal(0, np.sqrt(rho), users)
    noise = rng.normal(0, np.sqrt(1 - rho), (users, sessions))
    y = user_effect[:, None] + noise
    sess_p = ttest_ind(y[arm].ravel(), y[~arm].ravel(), equal_var=False).pvalue
    user_p = ttest_ind(y[arm].mean(1), y[~arm].mean(1), equal_var=False).pvalue
    return sess_p, user_p


print("Randomization is by user. There is no treatment effect. Each user")
print("contributes %d sessions." % sessions)
print()
print("%-30s %22s %22s"
      % ("within-user correlation", "analyzed by session", "analyzed by user"))
for rho in (0.0, 0.1, 0.3, 0.6):
    s = np.array([one_trial(rho) for _ in range(reps)])
    print("%-30.1f %21.1f%% %21.1f%%"
          % (rho, (s[:, 0] < 0.05).mean() * 100, (s[:, 1] < 0.05).mean() * 100))
print()
print("Both columns test the same data with the same nominal 5% level.")
# Randomization is by user. There is no treatment effect. Each user
# contributes 12 sessions.
#
# within-user correlation           analyzed by session       analyzed by user
# 0.0                                              5.7%                   5.5%
# 0.1                                             18.0%                   5.7%
# 0.3                                             35.5%                   5.9%
# 0.6                                             49.1%                   4.7%
#
# Both columns test the same data with the same nominal 5% level.

With no treatment effect and a nominal 5% level, analyzing by session gives a false positive rate of 18.0% at a within-user correlation of 0.1 and 49.1% at 0.6. Analyzing the same data by user stays between 4.7% and 5.9% throughout.

Bakshy and Eckles discuss how ignoring dependence can understate uncertainty. For equal clusters of 12 sessions with a common within-user correlation rho, the simple grand-mean design effect is 1 + 11rho. Thus effective sample size is about 5.71N sessions at rho = 0.1 and 1.58N at rho = 0.6 for N users, rather than 12N. This approximation is not a universal formula for unequal cluster sizes or ratio metrics.

The effect can move while you measure it

A fixed horizon estimates the effect averaged over its observation period. Novelty or learning can make that quantity differ from the steady-state effect without invalidating the finite-period comparison.

import numpy as np

rng = np.random.default_rng(2718)
base = 0.060
lift_day1, lift_long, tau = 0.090, 0.008, 7.0
per_day = 9000
hurdle = 0.020

days = np.arange(1, 57)
lift = lift_long + (lift_day1 - lift_long) * np.exp(-(days - 1) / tau)

print("The treatment lifts conversion by %.1f%% on day 1 and settles at %.1f%%."
      % (lift_day1 * 100, lift_long * 100))
print("Base rate %.1f%%, %s users a day per arm, ship if the lift beats %.1f%%."
      % (base * 100, "{:,}".format(per_day), hurdle * 100))
print()
print("%-14s %16s %16s %16s %12s"
      % ("window", "true average", "measured", "long-run truth", "decision"))
for w in (3, 7, 14, 28, 56):
    true_avg = lift[:w].mean()
    c = rng.random((w, per_day)) < base
    t = rng.random((w, per_day)) < base * (1 + lift[:w])[:, None]
    measured = t.mean() / c.mean() - 1
    print("%-14s %15.2f%% %15.2f%% %15.2f%% %12s"
          % ("%d days" % w, true_avg * 100, measured * 100, lift_long * 100,
             "ship" if measured > hurdle else "drop"))
print()
print("Each row is a separate simulation; the hurdle is not a significance test.")
# The treatment lifts conversion by 9.0% on day 1 and settles at 0.8%.
# Base rate 6.0%, 9,000 users a day per arm, ship if the lift beats 2.0%.
#
# window             true average         measured   long-run truth     decision
# 3 days                    7.96%           10.03%            0.80%         ship
# 7 days                    6.36%            6.47%            0.80%         ship
# 14 days                   4.60%            3.61%            0.80%         ship
# 28 days                   2.96%            2.75%            0.80%         ship
# 56 days                   1.90%            1.02%            0.80%         drop
#
# Each row is a separate simulation; the hurdle is not a significance test.

The code generates separate synthetic experiments for five durations, with point estimates of 10.03%, 6.47%, 3.61%, 2.75% and 1.02%. Four exceed a 2% point-estimate shipping hurdle even though the generator’s long-run effect is 0.8%. No significance test or confidence interval is computed in this block; it is not five valid hypothesis-test decisions on one shared trajectory.

Predeclare the target period, any burn-in exclusion and the analysis horizon. Plot daily effects to understand time variation, but do not extend or trim a fixed-horizon experiment after inspecting outcomes while retaining its original error guarantee. Adaptive stopping needs an appropriate sequential design or an explicitly exploratory follow-up.

A useful readout checks assignment and logging, accounts for dependence at the randomization level, and states the period over which the effect is estimated. Those checks complement the predeclared analysis and stopping rule described in the A/B testing entry.

References

• Fabijan et al. (2019), Diagnosing Sample Ratio Mismatch: author publication abstract
• Bakshy and Eckles (2013), Uncertainty in Online Experiments with Dependent Data
• Numerical scenarios, chosen rates and simulations in this entry are illustrative calculations by the author, not measurements reported by these sources.