Sample Ratio Mismatch

A sample ratio mismatch (SRM) occurs when the number of units observed in each arm of an experiment differs from the split that was designed, by more than chance would allow. If a rollout assigns 50 percent of users to control and 50 percent to a candidate, and the logs show 51,000 against 49,000, something other than random assignment is shaping which units get recorded. SRM is a data-quality alarm, not a result: it says the groups being compared are probably not comparable.

Why it matters

Randomization is what lets a difference in outcomes be attributed to the change under test. If the mechanism that drops or adds units depends on the arm, the surviving units differ systematically. Suppose the candidate crashes before logging for some sessions. The logged candidate sessions are the ones that survived, so their success rate looks better than it is. The outcome metric gives no sign of the problem; only the counts do.

The check

The standard check is a chi-square goodness-of-fit test of observed counts against the expected counts under the planned split. For two arms with total \(n\), planned shares \(p_1, p_2\), and observed counts \(n_1, n_2\):

\[ \chi^2 = \sum_i \frac{(n_i – n p_i)^2}{n p_i}. \]

The statistic is the squared gap between observed and expected counts, scaled by the expected count. With \(a\) arms it is compared to a chi-square distribution with \(a-1\) degrees of freedom. For two arms (one degree of freedom) the p-value is \(\mathrm{erfc}(\sqrt{\chi^2/2})\).

Computed for a planned 50/50 split:

control / candidate	\(\chi^2\)	p-value
50,000 / 50,000	0.0	1.0
50,500 / 49,500	10.0	0.0016
50,800 / 49,200	25.6	4.2e-7
5,080 / 4,920	2.56	0.11

The last row has the same 50.8 percent share as the third, but with a tenth of the traffic the deviation is within chance. SRM checks need volume: a one-percent imbalance is alarming at 100,000 units and unremarkable at 10,000.

Choosing the threshold

Because the check runs on every experiment, a conventional 0.05 cutoff would flag one healthy experiment in twenty. Practitioners commonly use a much stricter threshold, such as p below 0.001, so that an alert is rare and means something. When SRM fires, results from the experiment are not trusted until the cause is found, and the comparison is not rescued by reweighting.

Typical causes

• Logging that depends on the arm: one path crashes, times out, or redirects before the exposure event is recorded.
• Assignment that is not purely random: bucketing by an attribute correlated with behavior, a bot filter applied only to some arms, or a cache that serves one arm more.
• Triggering conditions that differ between arms, such as counting a user only when a feature is rendered and the candidate renders it more or less often.
• Carryover from earlier experiments, or a ramp in which the planned share changed during the period analyzed.
• Mid-experiment assignment changes, or merging data from periods with different splits.

The check should be run on the counts of units that enter the analysis, using the same filters, so that it catches problems introduced by the pipeline as well as by assignment.

Limits

A passing check does not prove the arms are comparable; it only means the counts match the plan. Mismatches in composition can occur with matching counts. A failing check does not identify the cause; it directs investigation to logging, assignment, and filtering. With more than two arms, the same test applies with more degrees of freedom, and the p-value comes from a chi-square distribution with \(a-1\) degrees of freedom.