Online Experimentation Platforms

An online experimentation platform is the infrastructure a company builds or buys so that controlled experiments — A/B tests, multivariate tests, bandits — can be run on a digital product as routine work rather than as projects. It lets product managers, data scientists and engineers assign users to variants at random, collect the data automatically, analyze it with methods chosen in advance, and enforce guardrails so that a bad variant does not reach many people.

Six components do that work. Setup and randomization assigns users — by cookie, device ID or account — and keeps the assignment stable, so the same user always sees the same variant. Data collection and logging tracks the metrics consistently across pipelines. A metrics framework holds predefined KPIs and guardrails, from which an experimenter picks a primary and some secondary ones. A statistical engine supplies frequentist p-values and confidence intervals under fixed-horizon or sequential designs, Bayesian posteriors and credible intervals, and variance reduction. A dashboard reports significance, effect sizes and intervals over time. And governance flags negative effects before launch and keeps the whole thing inside legal and ethical limits.

Modern platforms add sequential testing so that checking early does not inflate the error rate, Bayesian reporting such as the probability that B beats A, multi-armed bandits for adaptive allocation, heterogeneous treatment effect analysis by segment, integration with personalization models, and the ability to run hundreds of experiments at once.

That last capability is where a platform stops being a bigger version of one experiment and becomes a different statistical object.

Three things that only appear at platform scale

import numpy as np
from scipy.stats import norm

alpha = 0.05
print(f"{'experiments':>13}{'metrics each':>14}{'tests run':>12}"
      f"{'expected false flags':>22}{'P(any metric flags)':>23}")
for n_exp in (1, 20, 200):
    for n_met in (1, 20):
        tests = n_exp * n_met
        print(f"{n_exp:>13}{n_met:>14}{tests:>12}{tests * alpha:>22.1f}"
              f"{1 - (1 - alpha) ** n_met:>23.4f}")

rng = np.random.default_rng(0)
runs, n = 4_000, 4_000
print(f"{'correlation':>13}{'sd of the estimate':>21}{'vs no adjustment':>19}"
      f"{'equivalent traffic':>21}")
base = None
for rho in (0.0, 0.3, 0.5, 0.7, 0.9):
    pre = norm.ppf(rng.random((runs, n)))
    post = rho * pre + np.sqrt(1 - rho ** 2) * norm.ppf(rng.random((runs, n)))
    assign = np.tile([0, 1], n // 2)
    adj = post - rho * pre
    theta = adj[:, assign == 1].mean(axis=1) - adj[:, assign == 0].mean(axis=1)
    sd = theta.std(ddof=1)
    if base is None:
        base = sd
    print(f"{rho:>13.1f}{sd:>21.6f}{sd / base:>19.4f}{(base / sd) ** 2:>21.2f}x")

print(f"{'interaction':>13}{'true effect of A':>19}{'measured':>11}{'bias':>10}"
      f"{'measured B':>13}")
runs2, m = 1_000, 20_000
for inter in (0.00, 0.02, 0.05):
    ua = rng.random((runs2, m)) < 0.5
    ub = rng.random((runs2, m)) < 0.5
    base_rate = 0.10 + 0.03 * ua + 0.02 * ub + inter * (ua & ub)
    y = rng.random((runs2, m)) < base_rate
    est_a = (y * ua).sum(axis=1) / ua.sum(axis=1) - (y * ~ua).sum(axis=1) / (~ua).sum(axis=1)
    est_b = (y * ub).sum(axis=1) / ub.sum(axis=1) - (y * ~ub).sum(axis=1) / (~ub).sum(axis=1)
    print(f"{inter:>13.2f}{0.03:>19.4f}{est_a.mean():>11.4f}"
          f"{est_a.mean() - 0.03:>10.4f}{est_b.mean():>13.4f}")
#   experiments  metrics each   tests run  expected false flags    P(any metric flags)
#             1             1           1                   0.1                 0.0500
#             1            20          20                   1.0                 0.6415
#            20             1          20                   1.0                 0.0500
#            20            20         400                  20.0                 0.6415
#           200             1         200                  10.0                 0.0500
#           200            20        4000                 200.0                 0.6415
#   correlation   sd of the estimate   vs no adjustment   equivalent traffic
#           0.0             0.031236             1.0000                 1.00x
#           0.3             0.030241             0.9681                 1.07x
#           0.5             0.027232             0.8718                 1.32x
#           0.7             0.022592             0.7233                 1.91x
#           0.9             0.013609             0.4357                 5.27x
#   interaction   true effect of A   measured      bias   measured B
#          0.00             0.0300     0.0298   -0.0002       0.0200
#          0.02             0.0300     0.0402    0.0102       0.0302
#          0.05             0.0300     0.0550    0.0250       0.0447

Reading the three blocks

The first is why guardrails need a correction policy rather than a threshold. A single experiment watching twenty guardrail metrics at 5% trips at least one of them 64% of the time on data containing nothing, and two hundred such experiments produce an expected two hundred false metric flags, spread across about 128 experiments under the independent-metric setup. Every one costs an investigation. The answer is not to watch fewer metrics — guardrails exist precisely to catch what nobody predicted — but to apply a multiple-comparisons policy at the platform level and to state it on the dashboard.

The second prices variance reduction. Using a pre-period measurement as a covariate — the CUPED idea — shrinks the standard error in proportion to how well the past predicts the present. At a correlation of 0.7 the same precision arrives with half the traffic, and at 0.9 with a fifth. Nothing about the experiment changes; the estimator simply stops re-measuring what was already known. For a platform this is the cheapest sample size available, which is why it is a standard feature rather than a refinement.

The third block concerns interaction between treatments assigned to the same users, not necessarily interference between users. In its additive-plus-interaction model, if the other experiment assigns treatment with probability q, the marginal effect includes q times the interaction. With q = 0.5 that is half. Randomization identifies the effect under that traffic mixture; it is not the standalone effect with the other treatment absent. Joining assignment logs allows a factorial analysis to estimate the interaction, or separate traffic can target standalone effects.

What exists, and why it is usually built

On the commercial side, Optimizely is the best-known SaaS experimentation tool, Adobe Target serves enterprise A/B testing and personalization, VWO aims at marketing teams, and Split.io and LaunchDarkly come at it from feature flagging. Google Optimize was the free entry point until it was withdrawn on 30 September 2023; Google did not replace it, and directs users to third-party A/B testing tools integrated with Google Analytics instead — the current state of Google’s experimentation offerings is worth reading on its own.

Large technology companies mostly build their own: Microsoft’s ExP platform behind Bing, Office and Windows; Airbnb’s experiment reporting framework; Uber’s and Netflix’s experimentation infrastructure; LinkedIn’s XLNT; Meta’s PlanOut and the infrastructure around it. The reason is usually not statistics. It is that the randomization unit, the metric definitions and the guardrails are specific to the product, and that the three effects measured above — correction policy, variance reduction, interference — have to be handled against that product’s own traffic.

References

• Google, “Sunset of Google Optimize” (Optimize Help). support.google.com
• SciPy documentation, scipy.stats.norm. docs.scipy.org