Multivariate Test (MVT)
A multivariate test varies more than one element of an experience in a single experiment. Kohavi and colleagues define it that way — “an experiment that includes more than one factor is often called a MultiVariable test (MVT)” — and note in a footnote that the same thing is also called multivariate testing, which is the more common name in marketing tools. Three elements with two versions each gives \(2 \times 2 \times 2 = 8\) combinations; five elements gives \(2^5 = 32\).
The appeal is doing in one experiment what would otherwise be several, and learning how the elements interact. Both parts of that appeal deserve examination, and the first survives it better than the second.
It does not cost more traffic
The most widely repeated claim about multivariate tests is that more combinations need more traffic. For the main effects — the question of whether each element helps — that is wrong, and Kohavi and colleagues state the opposite plainly: “When running an eight run MVT or a full factorial the power to detect a difference for any main effect is the same.”
The reason is in how a factorial design allocates. Each element is set to one version in half the combinations and the other version in the other half, so the comparison for that element uses the entire sample — not one eighth of it. The other elements vary underneath both halves and average out. Eight combinations do not make eight small experiments; they make three full-sized ones sharing the same traffic.
Two things do reduce power, and the same source names them. Giving a factor more than two levels splits the comparison for that factor, which cuts its effective sample whether the design is an MVT or an A/B test. And allocating less than half the population to a treatment costs precision: “It is especially important for treatments in an MVT to have the same percentage of the population as the Control.”
How many comparisons is this
Where the design does become expensive is in what gets tested at the end. An eight-arm test compared against a control is seven comparisons, and if each is run at the 5 percent level and the comparisons were independent the chance that at least one is significant when nothing is happening would be \(1 - 0.95^7 = 0.302\). They are not independent, though: all seven share the same control group, so the events are positively correlated and the true figure is lower — a null simulation with equal-sized normal arms puts it near 0.229. The independent calculation is a rough upper reference, not the design’s error rate. For a common-control design the right tools account for the shared arm: Dunnett’s procedure, available as scipy.stats.dunnett, or a joint distribution over the comparisons. Comparing every arm with every other is \(\binom{8}{2} = 28\) comparisons, and the arithmetic gets worse quickly.
This is the many-arms form of the problem that significance level and Type I error describe, and its sibling is the repeated-looks form under risk of peeking. The first step is the one teams skip: counting how many comparisons were actually made, including the ones made informally while scanning a results table. A correction then follows — and corrections are not free, since as Kohavi and colleagues note they “basically equate to increasing the 95% confidence level and thus reducing the statistical power”.
Testing the main effects rather than the combinations avoids most of this. Three factors give three planned comparisons, not seven or twenty-eight, and each uses the full sample.
Interactions are rarer than expected
The other justification for an MVT is that it reveals how elements combine — that a headline works only with a particular image. Kohavi and colleagues report the opposite from practice twice over: “Large interactions between factors are actually rarer than most people believe, unless they are already known”, and “strong interactions are rare in practice, and we believe this concern is overrated”.
That matters for design choice, because estimating interactions well is exactly what a reduced design gives up. Traditional MVTs generally use fractional factorial or Plackett–Burman designs — subsets of the full factorial, popularized as Taguchi designs — and the warning attached is that “the user must be careful to choose a design that will have sufficient resolution to estimate the main effects and interactions that are of interest”. A design chosen for traffic efficiency may be unable to separate an interaction from a main effect at all, and that limitation is a property of the design rather than something the analysis can recover.
What to run instead
For online experiments Kohavi and colleagues prefer two alternatives to the traditional MVT, and say so: “We believe the two alternatives presented above are better than the traditional MVT approach for online experiments.”
| Approach | How it runs | Choose it when |
|---|---|---|
| Traditional MVT | A fractional factorial or Plackett–Burman design, launched at once | The design’s resolution genuinely covers the interactions you need, and everything is ready together |
| Concurrent univariate tests | Each factor as its own experiment, randomized independently, running at the same time | Interactions matter; independent randomization effectively recovers a full factorial |
| Overlapping experiments | Each factor launched when it is ready, overlapping where they happen to | Speed matters most; interactions can still be estimated where two tests overlap enough |
The practical drawback of the first row is scheduling: the test cannot start until every factor is built. The second and third rows start each factor when it is ready, which in an environment shipping continuously is worth a great deal. One further piece of advice from the same source is worth keeping for teams new to experimentation: “we believe the first experiment one does should be an A/B test mainly due to the complexity of testing more than one factor in the same test” — see A/B testing.
References
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.