Kolmogorov-Smirnov Test (KS Test)

The Kolmogorov–Smirnov test compares two distributions by the largest vertical gap between their cumulative distribution functions:

\[ D_{n,m} = \sup_x |F_n(x) - G_m(x)| \]

It makes no assumption about the shape of either distribution, which is both its appeal and the source of its blind spots.

One number from one location

import numpy as np
from scipy.stats import ks_2samp, norm

rng = np.random.default_rng(0)
a = rng.normal(0, 1, 2_000)
b = rng.normal(0.2, 1, 2_000)
res = ks_2samp(a, b)
grid = np.sort(np.concatenate([a, b]))
ecdf = lambda s, x: np.searchsorted(np.sort(s), x, side="right") / len(s)
gap = np.abs(ecdf(a, grid) - ecdf(b, grid))
print(f"    scipy reports D = {res.statistic:.6f}, p = {res.pvalue:.6f}")
print(f"    computed directly from the two ECDFs: {gap.max():.6f}")
print(f"    the gap is largest at x = {grid[gap.argmax()]:.4f}, near the middle")
"""D is a supremum over x, so one location decides the whole statistic."""

print(f"{'distribution drawn from':>26}{'rejection rate at 0.05':>25}")
for name, draw in (("normal", lambda n: rng.normal(size=n)),
                   ("exponential", lambda n: rng.exponential(size=n)),
                   ("heavy-tailed t(2)", lambda n: rng.standard_t(2, size=n))):
    rate = np.mean([ks_2samp(draw(200), draw(200)).pvalue < 0.05 for _ in range(600)])
    print(f"{name:>26}{rate:>25.4f}")

print(f"{'difference':>34}{'D':>10}{'p-value':>12}")
n = 4_000
for name, x, y in (("means differ by 0.2", rng.normal(0, 1, n), rng.normal(0.2, 1, n)),
                   ("same center, 20% wider", rng.normal(0, 1, n), rng.normal(0, 1.2, n)),
                   ("identical but for the far tail",
                    rng.normal(0, 1, n),
                    np.concatenate([rng.normal(0, 1, n - 40), rng.normal(6, 0.5, 40)]))):
    r = ks_2samp(x, y)
    print(f"{name:>34}{r.statistic:>10.6f}{r.pvalue:>12.6f}")

print(f"{'sample size each':>18}{'D':>10}{'p-value':>14}")
for m in (100, 1_000, 10_000, 100_000):
    r = ks_2samp(rng.normal(0, 1, m), rng.normal(0.02, 1, m))
    print(f"{m:>18}{r.statistic:>10.6f}{r.pvalue:>14.8f}")
#     scipy reports D = 0.085000, p = 0.000001
#     computed directly from the two ECDFs: 0.085000
#     the gap is largest at x = 0.3844, near the middle
#    distribution drawn from   rejection rate at 0.05
#                     normal                   0.0317
#                exponential                   0.0317
#          heavy-tailed t(2)                   0.0367
#                         difference         D     p-value
#                means differ by 0.2  0.101250    0.000000
#             same center, 20% wider  0.044500    0.000725
#     identical but for the far tail  0.014000    0.828022
#   sample size each         D       p-value
#                100  0.180000    0.07822116
#               1000  0.046000    0.24068202
#              10000  0.015000    0.21055778
#             100000  0.008900    0.00072171

The first block confirms what the statistic is by computing it two ways: SciPy reports \(D = 0.085000\), and the maximum gap taken directly from the two step functions is the same number. The gap is largest at \(x = 0.3844\), near the center of the data.

That is the first thing to hold on to. \(D\) is a supremum, so a single location decides the entire statistic. Everything the two distributions do elsewhere is discarded.

The second block demonstrates the distribution-free property that makes the test attractive: drawing both samples from a normal, an exponential, or a heavy-tailed \(t\) gives the same rejection behavior, because the null distribution of \(D\) does not depend on the underlying shape. All three sit a little below 0.05 rather than on it, since the two-sample statistic can only take finitely many values at this sample size and the test is conservative as a result.

Where it looks

The third block is the practical shape of that supremum. A shift in means gives \(D = 0.101250\). A 20% change in spread gives 0.044500, detected but weakly. And a sample contaminated in the far tail — 1% of the data moved out to six standard deviations — gives \(D = 0.014000\) with a p-value of 0.828022: not detected at all.

A small tail mass can create only a small absolute CDF gap, even when its values move far away. KS can detect that gap with enough observations; it is not blind to tails. For a decision specifically about extreme losses, assess tail probabilities or quantiles, or a tail-weighted procedure with its own assumptions, alongside the overall CDF difference.

The p-value reports headcount

The last block holds the true difference fixed at 0.02 standard deviations and grows the samples. \(D\) stays tiny throughout — 0.180000, 0.046000, 0.015000, 0.008900 — while the p-value falls from 0.078 to 0.00072. Only the largest sample rejects, and it rejects a difference that shrank rather than grew.

This matters most where KS is most used. A drift monitor comparing production traffic against a training set has very large samples by construction, so it can detect very small nonzero differences. Features whose distributions are unchanged do not become certain rejections merely because the samples grow. The p-value answers whether a difference exists; \(D\) is the one that says how big it is, and a threshold on \(D\) is what a monitor should alert on. The same distinction applies to statistical significance generally.

Two further limits are worth stating. The one-sample form requires the reference distribution to be fully specified in advance — estimating its parameters from the same data invalidates the standard p-values, which is why testing normality by fitting a mean and variance first needs the Lilliefors correction instead. And the test assumes continuous data; ties, which discrete or rounded data produce in quantity, make the usual null distribution approximate rather than exact.

References

• SciPy documentation, scipy.stats.ks_2samp. docs.scipy.org