A/B Testing
An A/B test is a randomized experiment: units are assigned to a control or a treatment by chance, and the outcome difference estimates the effect of assignment, subject to sampling uncertainty and the design conditions. The reason to bother with the randomization — rather than comparing people who happened to see the new version against people who did not — is precise, and it is worth stating in the language of causal inference.
Estimating a causal effect from observational data “requires two elements: data and identifiability conditions,” and those conditions are assumptions you cannot check from the data. In a randomized experiment, by contrast, “the identifiability conditions hold by design.” The quoted identification result concerns an ideal randomized experiment. Random assignment makes groups comparable in expectation; a finite-sample difference still contains random variation. Interference, missing outcomes, nonadherence, and measurement differences require separate attention.
What randomization does and does not settle
| Question | Does randomization answer it? |
|---|---|
| Does assignment support estimating a causal effect? | Yes under the experimental conditions; the observed difference also includes sampling variation and may be affected by design departures |
| Is the difference big enough to see? | No — that is power and sample size |
| Is it big enough to act on? | No — a business judgment against an effect size; see trivial effects |
| Does the metric measure what we care about? | No — see metric definition |
| Will the effect hold for other users, later? | No — the estimate is for the population you randomized, in the period you ran |
Randomization addresses confounding by the assignment mechanism; it does not eliminate every assumption or implementation failure. Specify the assignment effect, analyze the assigned groups, quantify uncertainty, and check whether interference, attrition, measurement, or nonadherence changes the interpretation. An effect of receiving treatment is a different estimand from an effect of being assigned it.
Inference must respect the assignment and dependence structure
When assignment is per user, events from one user cannot simply be treated as independent observations. One approach aggregates to user-level outcomes. Another retains event-level observations but uses a suitable model and variance estimator, such as cluster-robust inference by user with enough independent clusters. Choose the estimand first: equal weighting of users and equal weighting of requests answer different questions.
The same principle applies to assignment by store, city, school, or another cluster. Individual-level analysis is possible when it accounts for the assignment and dependence structure; counting individual rows as independent is the error. Few clusters require additional care, and aggregation by itself does not prove independence.
Four ways the design breaks in practice
- Contamination between arms. If treated users can affect control users — a shared marketplace, a social feed, a price both groups see, a support queue both wait in — then the control is no longer a clean counterfactual and the difference understates or overstates the effect. Randomizing a larger unit is the usual remedy.
- Selection after assignment. Filtering to users who completed something, or who were active for a week, reintroduces exactly the problem randomization removed — the filter can depend on the treatment. Compare everyone who was assigned.
- Stopping when it looks good. Deciding to stop based on the result changes the error rate of the procedure, whatever the final p-value says. Fix the horizon in advance or use a method designed for repeated looks — see risk of peeking and stopping rules.
- Choosing the metric afterward. With enough metrics and segments, something will look significant; that is the multiple-comparison arithmetic in Type I error, not a discovery. Name the primary metric before the test runs.
Note what the first two have in common: both are ways the assignment stops being the only difference between the groups. That is the property to protect, and it is protected by the design rather than by the analysis.
Reading the result
Report the difference with an interval, not a verdict. A confidence interval for the lift carries both the magnitude and the precision, and it makes an uninformative test visible: a range from a 4% loss to a 5% gain says the experiment could not distinguish the outcomes you care about, where “not significant” hides that. The threshold logic itself is in hypothesis testing and statistically significant.
Two further limits on the reading. The estimate is an average over the randomized population, so it can be positive overall while negative for a segment — estimating those differences deliberately is what uplift models are for, and doing it by slicing after the fact is the multiple-comparison problem again. And a test measures the period it ran in, so a novelty response or a seasonal window is part of what you measured, not noise around it.
Where an experiment is impossible — the change is already live everywhere, or randomizing would be unethical or uneconomic — the fallback is an observational design with stated assumptions, which is a weaker claim rather than an equivalent one. See causal inference and causal impact.
References
- Miguel A. Hernán & James M. Robins, Causal Inference: What If (Chapman & Hall/CRC, 2020), chapters 1, 3 and 16.
- statsmodels, Cluster-robust covariance
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.