McNemar Test

The McNemar test compares two classifiers, systems or releases on the same set of cases when each case gets a pass or fail from both. Because the two outcomes on one case are linked, the comparison should use only the cases where the systems disagree.

Setup

Run systems A and B on the same \(n\) cases and tabulate the pairs:

	B passes	B fails
A passes	\(a\)	\(b\)
A fails	\(c\)	\(d\)

Cases in \(a\) and \(d\) are concordant: both passed or both failed. They say nothing about which system is better. Cases in \(b\) (only A passes) and \(c\) (only B passes) are the discordant pairs.

The test

If the systems are equally good, then among the \(m=b+c\) discordant cases, each is equally likely to favor A or B. The count \(b\) then follows a binomial distribution with \(m\) trials and probability 1/2. The exact two-sided p-value is

\[ p = \min\!\left(1,\; 2\sum_{j=0}^{\min(b,c)} \binom{m}{j} 2^{-m}\right). \]

This is a sign test on the discordant pairs. A common large-sample form is the statistic \((|b-c|-1)^2/(b+c)\), compared with a chi-square distribution with 1 degree of freedom; the exact form is better when \(m\) is below a few dozen.

Worked example (hypothetical)

Take 100 cases: both pass on 70, only A passes on 4, only B passes on 12, both fail on 14. A's pass rate is 0.74 and B's is 0.82.

• Exact test: \(m=16\), \(\min(b,c)=4\), p = 0.077.
• Chi-square form with continuity correction: statistic 3.06, p = 0.080.
• A two-proportion z-test that ignores pairing: z = 1.37, p = 0.172.

The unpaired test is much less sensitive because it treats the 100 cases for A and the 100 cases for B as independent samples, so case-to-case difficulty adds noise. The paired test removes that noise: an easy case is easy for both systems and cancels out.

The count of discordant pairs, not the number of cases, sets the power. With only A 10 and only B 18 (a gap of 8), p = 0.185. With only A 2 and only B 10 (also a gap of 8, but fewer disagreements), p = 0.039. The same gap is more convincing when the systems disagree less often overall, because there is less room for chance.

Assumptions and limits

• Each case must be scored by both systems, and cases must be independent of each other. Cases built from the same source document or the same user are not independent, and the p-value will be too small.
• It compares overall pass rates. It does not weight cases. If the test set oversamples some strata, the result speaks about the test set, not about traffic.
• It handles binary outcomes. For graded scores, use a paired t-test, a Wilcoxon signed-rank test, or a paired bootstrap.
• A non-significant result does not show that systems are equal; with few discordant pairs the test has little power.
• The list of discordant cases is itself useful output: a short set of examples where behavior changed, each of which a person can read.

Related ideas

Cochran's Q test extends the idea to three or more systems on the same cases. The sign test is the general form for any paired comparison reduced to "which was better".