DeLong’s Test

DeLong’s test is a non-parametric test for comparing the ROC-AUCs of two correlated models, proposed by DeLong, DeLong and Clarke-Pearson in 1988. It answers the question that follows every model comparison: is model A’s AUC significantly better than model B’s?

It is needed because an AUC is a random variable. It depends on which cases happened to be in the test set, so the gap \(\text{AUC}_A - \text{AUC}_B\) is not evidence by itself. And because two models are almost always evaluated on the same test set, their AUCs are correlated — a test set that flatters one model tends to flatter the other. DeLong’s test supplies the standard error of each AUC, the standard error of their difference, and from those a \(z\) and a p-value:

\[ z = \frac{\text{AUC}_1 - \text{AUC}_2}{\text{SE}(\text{AUC}_1 - \text{AUC}_2)} \]

The mechanism is a U-statistic argument. An AUC is the probability that a random positive is ranked above a random negative, so it can be written as an average over all positive–negative pairs. DeLong’s derivation computes the variances and covariances of those pairwise comparisons — the structural components, one per case — and assembles them into a covariance matrix over the models.

The estimator, and what the correlation is worth

import numpy as np
from scipy.stats import norm
from sklearn.metrics import roc_auc_score

def delong_cov(y, scores):
    """DeLong's estimator: AUC is a two-sample U-statistic, so its variance
       comes from the structural components V10 and V01."""
    pos, neg = scores[:, y == 1], scores[:, y == 0]
    m, n = pos.shape[1], neg.shape[1]
    psi = (pos[:, :, None] > neg[:, None, :]).astype(float) \
        + 0.5 * (pos[:, :, None] == neg[:, None, :])
    v10, v01 = psi.mean(axis=2), psi.mean(axis=1)
    auc = v10.mean(axis=1)
    s10 = np.cov(v10, ddof=1)
    s01 = np.cov(v01, ddof=1)
    return auc, np.atleast_2d(s10) / m + np.atleast_2d(s01) / n

rng = np.random.default_rng(0)
n_pos, n_neg = 300, 700
y = np.r_[np.ones(n_pos), np.zeros(n_neg)]
signal = np.r_[norm.ppf(rng.random(n_pos)) + 1.0, norm.ppf(rng.random(n_neg))]
a = signal + 0.55 * norm.ppf(rng.random(n_pos + n_neg))
b = signal + 0.95 * norm.ppf(rng.random(n_pos + n_neg))
"""Two models scored on the same test set. They share the signal, so their
   errors - and therefore their AUCs - move together."""

auc, cov = delong_cov(y, np.vstack([a, b]))
diff = auc[0] - auc[1]
se_paired = np.sqrt(cov[0, 0] + cov[1, 1] - 2 * cov[0, 1])
se_naive = np.sqrt(cov[0, 0] + cov[1, 1])
print(f"  model A AUC {auc[0]:.6f}   (sklearn {roc_auc_score(y, a):.6f})")
print(f"  model B AUC {auc[1]:.6f}   (sklearn {roc_auc_score(y, b):.6f})")
print(f"  difference  {diff:.6f}")
print(f"{'':>34}{'SE':>12}{'z':>10}{'p':>12}")
print(f"{'DeLong, accounting for pairing':>34}{se_paired:>12.6f}"
      f"{diff / se_paired:>10.4f}{2 * norm.sf(abs(diff / se_paired)):>12.6f}")
print(f"{'treating the AUCs as independent':>34}{se_naive:>12.6f}"
      f"{diff / se_naive:>10.4f}{2 * norm.sf(abs(diff / se_naive)):>12.6f}")
r = cov[0, 1] / np.sqrt(cov[0, 0] * cov[1, 1])
print(f"  the two AUC estimates correlate at {r:.4f}, so the naive standard error")
print(f"  is {se_naive / se_paired:.2f} times too large. The p-value it produces is"
      f" {2 * norm.sf(abs(diff / se_naive)) / (2 * norm.sf(abs(diff / se_paired))):.0f} times")

B = 4000
idx_p = rng.integers(0, n_pos, size=(B, n_pos))
idx_n = rng.integers(0, n_neg, size=(B, n_neg))
ap, an = a[:n_pos], a[n_pos:]
bp, bn = b[:n_pos], b[n_pos:]
d = np.empty(B)
for i in range(B):
    P, N = idx_p[i], idx_n[i]
    d[i] = ((ap[P][:, None] > an[N][None, :]).mean()
            - (bp[P][:, None] > bn[N][None, :]).mean())
print(f"    bootstrap SE of the difference {d.std(ddof=1):>10.6f}")
print(f"    DeLong SE                      {se_paired:>10.6f}")
print(f"    bootstrap 95% interval         "
      f"[{np.quantile(d, 0.025):.6f}, {np.quantile(d, 0.975):.6f}]")
print(f"    DeLong 95% interval            "
      f"[{diff - 1.96 * se_paired:.6f}, {diff + 1.96 * se_paired:.6f}]")

d0, se0 = 0.88 - 0.84, 0.015
z0 = d0 / se0
print(f"    difference {d0:.2f}, SE {se0:.3f} -> z = {z0:.4f},"
      f" p = {2 * norm.sf(abs(z0)):.6f}")
#   model A AUC 0.753429   (sklearn 0.753429)
#   model B AUC 0.702900   (sklearn 0.702900)
#   difference  0.050529
#                                             SE         z           p
#     DeLong, accounting for pairing    0.015541    3.2513    0.001149
#   treating the AUCs as independent    0.024856    2.0329    0.042066
#   the two AUC estimates correlate at 0.6105, so the naive standard error
#   is 1.60 times too large. The p-value it produces is 37 times
#     bootstrap SE of the difference   0.015711
#     DeLong SE                        0.015541
#     bootstrap 95% interval         [0.019909, 0.080670]
#     DeLong 95% interval            [0.020068, 0.080989]
#     difference 0.04, SE 0.015 -> z = 2.6667, p = 0.007661

Reading the output

The middle row of the first table is the point of the method. The two AUCs correlate at 0.61 because the models share a test set, and accounting for that shrinks the standard error of the difference by a factor of 1.60. Treating the two AUCs as independent and adding their variances produces a p-value about 37 times larger — still under 0.05 in this case, but the same mistake on a smaller difference discards a real finding.

The second block is the check worth running once. A case-resampling bootstrap of the same difference gives a standard error of 0.015711 against DeLong’s 0.015541, and interval endpoints that agree to three decimals. DeLong estimates the asymptotic covariance analytically. A paired, class-stratified bootstrap estimates sampling variability by resampling. They can agree closely but their finite-sample variance estimates are not identical; DeLong avoids Monte Carlo resampling noise.

The third block reproduces the worked example. A difference of 0.04 with a standard error of 0.015 gives \(z = 2.6667\) and \(p = 0.007661\), significant at the 1% level. The arithmetic is trivial; the standard error is not, and it is the only piece a side-by-side comparison of two AUC numbers does not have.

Where it applies, and what else is available

The natural uses are comparing two classifiers on the same test set, comparing a baseline against a candidate, and clinical work where a difference in discrimination has to be defended statistically rather than asserted.

The alternatives divide by cost and assumption. A bootstrap of the difference avoids a parametric score-distribution assumption but requires a suitable resampling design and handles quantities DeLong’s derivation does not cover, at the price of compute. The Hanley–McNeil approach is the earlier method and uses a cruder approximation to the correlation. Permutation validity depends on its null and exchangeability assumptions. Swapping paired model scores is not the same null as shuffling outcome labels: label shuffling tests absence of label-score association, generally stronger than equality of two nontrivial AUCs.

Two limits are worth stating. The test is asymptotic, so it needs a reasonable number of cases in the smaller class; with a few dozen positives the normal approximation is doing real work. And it is a test about one test set. A significant difference on this test set is not a claim about the next one, and if the two models were tuned on the same held-out data the comparison inherits that selection — the p-value does not know how many candidates were tried before these two.

References

• DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 44(3), 837–845. pubmed.ncbi.nlm.nih.gov
• scikit-learn documentation, sklearn.metrics.roc_auc_score. scikit-learn.org
• SciPy documentation, scipy.stats.norm. docs.scipy.org