Categorical Drift

Categorical drift is a change in the share of each category a feature takes between the training data and production. It is the same event as data drift on a numeric column, with one structural difference: there is no ordering, so there are no cumulative distributions to compare and the KS statistic has nothing to work with.

What replaces it is usually a list: a chi-square test, Cramér’s V, and PSI. Those three are not interchangeable, and the differences between them decide whether a monitor works.

A test and an effect size are not the same reading

The chi-square statistic for a two-row contingency table grows in proportion to the number of rows you feed it, because it is a sum over cells of squared count deviations. Cramér’s V divides that out. The code blocks in this entry run in order in one session, so later blocks reuse the imports and functions defined earlier.

\[ V = \sqrt{\frac{\chi^2}{N \cdot (\min(r, c) - 1)}} \]

import numpy as np
from scipy.stats import chi2_contingency
from scipy.stats.contingency import association


def draw(p, n, rng):
    edges = np.cumsum(p)
    idx = np.searchsorted(edges, rng.random(n))
    return np.array([(idx == k).sum() for k in range(len(p))])


def cramer_v(table):
    stat = chi2_contingency(table, correction=False)[0]
    return np.sqrt(stat / (table.sum() * (min(table.shape) - 1)))


base = np.array([0.50, 0.30, 0.15, 0.05])
curr = np.array([0.45, 0.32, 0.16, 0.07])
rng = np.random.default_rng(404)

print("%9s %11s %12s %10s %11s %11s %10s"
      % ("n / side", "chi-square", "p", "mean V", "V no drift",
         "V formula", "scipy V"))
for n, reps in ((200, 300), (2000, 300), (20000, 150), (100000, 60)):
    one = np.vstack([draw(base, n, rng), draw(curr, n, rng)])
    stat, p, _, _ = chi2_contingency(one, correction=False)
    drift = np.mean([cramer_v(np.vstack([draw(base, n, rng), draw(curr, n, rng)]))
                     for _ in range(reps)])
    flat = np.mean([cramer_v(np.vstack([draw(base, n, rng), draw(base, n, rng)]))
                    for _ in range(reps)])
    print("%9d %11.2f %12.2e %10.4f %11.4f %11.4f %10.4f"
          % (n, stat, p, drift, flat, cramer_v(one),
             association(one, method="cramer")))

 n / side  chi-square            p     mean V  V no drift   V formula    scipy V
      200        9.85     1.99e-02     0.0948      0.0827      0.1569     0.1569
     2000       15.72     1.29e-03     0.0632      0.0254      0.0627     0.0627
    20000      130.15     5.01e-28     0.0584      0.0083      0.0570     0.0570
   100000      743.77    6.76e-161     0.0593      0.0037      0.0610     0.0610

The shift is written into the script and never changes: the baseline shares are 0.50, 0.30, 0.15 and 0.05, and the current shares are 0.45, 0.32, 0.16 and 0.07 in every row of the table. For each sample size, one pair of samples gives the chi-square statistic and p-value; mean V averages Cramér’s V over repeated drifted samples and V no drift over repeated samples drawn from the baseline shares on both sides; the last two columns compute V from the formula and with SciPy. The p-value moves from 0.02 to 1e-160 across the table, which is a statement about how much data arrived rather than about the customers in it. That is the same failure the fixed thresholds on KS shift have, arriving by a different route.

Cramér’s V converges on about 0.059 instead, which is what an effect size is for. It is not free of the problem, though: at 200 rows a side the no-drift reading is 0.0827 and the real signal is 0.0948, so the shift is inside the noise. Quote V against the no-drift level for the window you have, not against a table of round numbers.

The category that was not there before

A new category has zero baseline share. That boundary changes how categorical drift measures must be computed and interpreted. The baseline shares are A 0.50, B 0.30, C 0.15, D 0.05 and new 0.00; the current shares are 0.48, 0.29, 0.14, 0.05 and 0.04, and the chi-square table is built from 50,000 rows on each side. psi sums \( (c - b)\ln(c/b) \) over categories after raising every share to a floor eps.

base = np.array([0.50, 0.30, 0.15, 0.05, 0.00])
curr = np.array([0.48, 0.29, 0.14, 0.05, 0.04])
n = 50000


def psi(b, c, eps=0.0):
    b, c = np.maximum(b, eps), np.maximum(c, eps)
    return float(np.sum((c - b) * np.log(c / b)))


print("%14s %12s" % ("floor epsilon", "PSI"))
for eps in (1e-8, 1e-6, 1e-4, 1e-3, 1e-2):
    print("%14.0e %12.4f" % (eps, psi(base, curr, eps)))

table = np.vstack([np.round(base * n), np.round(curr * n)]).astype(int)
stat, _, dof, exp = chi2_contingency(table, correction=False)
print("expected count for the new category   %.1f and %.1f" % (exp[0, -1], exp[1, -1]))
print("chi-square %.1f on %d degrees of freedom" % (stat, dof))
print("share of current traffic in the new category: %.1f%%" % (curr[-1] * 100))

 floor epsilon          PSI
         1e-08       0.6099
         1e-06       0.4257
         1e-04       0.2409
         1e-03       0.1457
         1e-02       0.0434
expected count for the new category   1000.0 and 1000.0
chi-square 2046.1 on 4 degrees of freedom
share of current traffic in the new category: 4.0%

PSI is a sum of \( (c - b)\ln(c/b) \) terms, so a zero baseline share with a positive current share gives an infinite term. A positive floor makes the displayed value finite, but its choice changes the result: the same data reads 0.6099 or 0.0434 depending on the chosen floor, and the conventional 0.1 and 0.25 lines both fall inside that range.

Chi-square survives, because its expected counts come from the column totals of the combined table rather than from the baseline row alone. So the three measures diverge exactly where the interesting thing happened. The number worth putting on a dashboard here is not a distance at all: it is the share of traffic landing in categories the model never saw (here 4.0%), reported on its own, with action determined by unknown-category handling, expected novelty, traffic share and validated impact.

Shares moved, but whose?

All three measures weight a category by how far its share moved. A model weights it by how much the answer depends on it. Those are different weightings, and they can disagree completely. The channel shares are retail 0.40, wholesale 0.30, online 0.20 and partner 0.10. The model adds +0.01, +0.00, -0.01 and +0.55 to a referral score by channel, on top of a base of 0.18 and noise with standard deviation 0.09, and a row is referred when its score exceeds 0.30. rate simulates 400,000 rows from a set of channel shares and returns the referral rate, while psi from the previous block measures the share movement.

base = np.array([0.40, 0.30, 0.20, 0.10])
effect = np.array([0.01, 0.00, -0.01, 0.55])
cutoff = 0.30

scenarios = {
    "big move, ordinary channels": np.array([0.15, 0.55, 0.20, 0.10]),
    "small move, the odd channel": np.array([0.38, 0.29, 0.20, 0.13]),
}


rng = np.random.default_rng(7)


def rate(shares, reps=400000):
    idx = np.searchsorted(np.cumsum(shares), rng.random(reps))
    score = 0.18 + effect[idx] + 0.09 * rng.normal(size=reps)
    return float((score > cutoff).mean())


base_rate = rate(base)
print("%-30s %10s %14s %14s" % ("drift", "PSI", "referral rate", "change"))
print("%-30s %10s %13.2f%% %14s" % ("no drift", "-", base_rate * 100, "-"))
for name, shares in scenarios.items():
    r = rate(shares)
    print("%-30s %10.4f %13.2f%% %12.2f pt"
          % (name, psi(base, shares), r * 100, (r - base_rate) * 100))

drift                                 PSI  referral rate         change
no drift                                -         18.57%              -
big move, ordinary channels        0.3967         18.13%        -0.44 pt
small move, the odd channel        0.0092         21.33%         2.76 pt

Twenty-five points of traffic move from retail to wholesale, two channels to which the model gives nearly the same effect (+0.01 and +0.00), and PSI reads 0.3967, about 1.6 times the 0.25 line usually called a major shift. The referral rate falls by only 0.44 points. Three points move into the one channel whose effect (+0.55) is very different, PSI reads 0.0092, and the referral rate goes up by 2.76 points of the whole portfolio.

A distribution distance cannot know which categories the model leans on, because it never sees the model. The cheap repair is to report both: the share movement for the drift log, and the shift in predicted rate for the people who have to staff the queue. When several categorical features move together rather than one at a time, investigate upstream joins, schema changes, population mix and external events; a macro shift is one possible common cause, not a diagnosis from co-movement alone.

References

• SciPy documentation, scipy.stats.chi2_contingency and scipy.stats.contingency.association.
• Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46(4), 1–37.