Two-Proportion Z-Test

The two-proportion z-test asks whether two independent groups have the same rate of some outcome: conversion in a control and a treatment group, defects from two production lines, recovery under two treatments. Let group 1 have \(x_1\) events out of \(n_1\) and group 2 have \(x_2\) out of \(n_2\), with observed rates \(\hat{p}_1 = x_1/n_1\) and \(\hat{p}_2 = x_2/n_2\). The NIST handbook builds the test from the normal approximation: “If the samples are reasonably large we can use the normal approximation to the binomial to develop a test similar to testing whether two normal means are equal.” The statistic is

\(z = (\hat{p}_1 - \hat{p}_2)\big/\sqrt{\hat{p}(1 - \hat{p})(1/n_1 + 1/n_2)}\), where \(\hat{p} = (x_1 + x_2)/(n_1 + n_2)\) is the pooled rate.

The pooling is not a shortcut. Under the null hypothesis the two groups share one rate, so the best estimate of that common rate uses both groups together, and the standard error of the difference is computed as if that single rate applied to both. For a two-sided test \(|z|\) is compared with the normal critical value \(z_{1-\alpha/2}\), 1.96 at the 5 percent level; for a one-sided alternative such as \(p_1 > p_2\), \(z\) is compared with \(z_{1-\alpha}\), 1.645. Equivalently the p-value is read from the normal distribution and compared with the significance level.

A worked example, and the interval that should go with it

Take an A/B test with illustrative numbers: the control converts 100 of 1,000 visitors, a rate of 10.0 percent, and the treatment converts 130 of 1,000, a rate of 13.0 percent. The pooled rate is 230 out of 2,000, or 0.115. The standard error under the null is \(\sqrt{0.115 \times 0.885 \times (1/1000 + 1/1000)} = 0.01427\), and the statistic is \(z = (0.100 - 0.130)/0.01427 = -2.10\). Its absolute value exceeds 1.96, and the two-sided p-value is 0.035, so at the 5 percent level the null hypothesis of equal rates is rejected.

from statsmodels.stats.proportion import proportions_ztest
stat, p = proportions_ztest(count=[100, 130], nobs=[1000, 1000])
# stat = -2.1027..., p = 0.0355

The statsmodels function uses the pooled standard error for the two-sample test, matching the formula above. Its documentation also records a fact worth knowing when different tools disagree: for a two-sided alternative the test gives the same p-value as the chi-square test on the two-by-two table, “since the chisquare is the distribution of the square of a standard normal distribution”. A chi-square test with a continuity correction, which some software applies by default, gives a slightly larger p-value than either.

The test says the rates differ; it does not say by how much. For that, take the difference in the direction of interest, treatment minus control, 0.030, and attach a confidence interval. The usual large-sample Wald interval uses the unpooled standard error, since it no longer assumes the rates are equal: \(\sqrt{0.10 \times 0.90/1000 + 0.13 \times 0.87/1000} = 0.01425\), and the 95 percent interval is \(0.030 \pm 1.96 \times 0.01425\), from 0.2 to 5.8 percentage points. Because the test pools and the interval does not, the two are not exact inversions of each other: they agree here and usually do, but an interval that excludes zero and a rejection at the same level are not guaranteed to coincide, and other interval constructions for a difference of proportions exist and are preferable for small counts. The point estimate is a 3-point lift, but the data are consistent with a lift that is almost nothing and with one nearly twice as large. That interval, not the p-value, is what the decision to ship should be argued from, and the practical significance entry covers what counts as enough.

When the normal approximation is not good enough

The approximation needs enough events and non-events in each group. The handbook’s criterion for the one-sample version is that the expected counts of successes and of failures should each be at least 5, and applying the same rule to each group of a two-sample test is a reasonable working threshold; with a base rate of 1 percent that means at least a few hundred observations per group before the test is trustworthy at all, and far more before it can detect a small lift. For small counts the handbook recommends the exact alternative: “The Fisher Exact Probability Test is an excellent nonparametric technique for analyzing discrete data (either nominal or ordinal), when the two independent samples are small in size.” Fisher’s test computes the probability of the observed table and of the more extreme ones directly from the hypergeometric distribution, with no approximation to break.

In an A/B test, the test is the easy part

• The unit. The formula counts \(n_1\) and \(n_2\) as independent observations. If the rows are sessions and the same user appears in many, the sessions are positively correlated, the true standard error is larger than the computed one, and the test rejects too often. Randomize at the unit the question is about, usually the user, and either count at that unit or use an analysis that respects the assignment, such as a cluster-aware standard error.
• The sample size is fixed in advance. Detecting a lift from 10 to 11 percent, or whatever the minimum detectable lift is, needs a sample size chosen from the base rate, the lift, \(\alpha\), and the desired power before the experiment starts.
• The horizon is fixed too. Recomputing the test every day and stopping at the first significant result changes the error rate of the whole procedure; see risk of peeking and stopping rules.
• One comparison per test. Several variants or several metrics multiply the chances of a false positive, which is the subject of multiple comparisons.

The wider design of an experiment, from assignment to reading the result, is covered under A/B testing, and the logic of rejecting or not rejecting under hypothesis testing.

References

• NIST/SEMATECH e-Handbook of Statistical Methods, section 7.3.3 How can we determine whether two processes produce the same proportion of defectives?
• NIST/SEMATECH e-Handbook of Statistical Methods, section 7.2.4 Does the proportion of defectives meet requirements?
• statsmodels documentation, statsmodels.stats.proportion.proportions_ztest