Label Drift

Label drift, also called label shift or prior probability shift, is a change in how often each outcome occurs: the distribution of the target, P(Y), differs between training and production or moves over time. A fraud model trained when 2% of transactions were fraudulent now sees 5%. A support-ticket classifier trained when 70% of tickets concerned billing now sees 40% billing and 60% technical issues.

Pure label shift assumes P(X|Y) stays unchanged while P(Y) changes: the feature distribution within each class remains the same, but their proportions differ. Covariate drift instead assumes stable P(Y|X) with changing P(X). These assumptions are useful models, not mutually exclusive names for every real incident. Under the definition of concept drift used in that entry—a change in P(Y|X)—pure label shift can also change the conditional outcome probabilities through Bayes’ rule.

When probabilities need updating

A change in P(Y) alone does not establish that a calibrated P(Y|X) is wrong: a changed input mix can change the overall rate while leaving the conditional probabilities intact. Under pure label shift, P(X|Y) stays fixed. For a binary outcome with overlap between classes, increasing the positive prior raises posterior probabilities that were strictly between zero and one. Perfectly separating features are an exception: if X determines Y, posterior probabilities of zero and one remain unchanged as class proportions move. Calibration therefore needs checking under the shift assumptions rather than being declared broken from the base rate alone.

Suppose a model gives a transaction probability 0.30 under a 2% fraud prior and the pure label-shift assumptions hold. Its old odds are 0.30/0.70. Multiply by the ratio of prior odds, (0.05/0.95)/(0.02/0.98), to obtain the odds at a 5% fraud prior. The updated probability is 0.525, or about 0.53. This transaction was below a 0.5 cutoff before and now exceeds it on the updated probability scale. The same probability cutoff can remain appropriate when decision costs are unchanged; an equivalent rule applied to the old scores uses a different cutoff. Reassess queue demand and capacity under the chosen rule.

For a fixed scoring function under pure label shift, the score distributions conditional on class stay the same, so the population ROC curve and ROC AUC remain unchanged. That preserves the old ranking performance, whatever its quality; it does not prove that the ranking is correct. Probability calibration and precision can change, and adjusting scores or thresholds can change which cases are selected. Finite-sample AUC estimates still fluctuate.

Measuring it when labels arrive late

When outcomes are known, the check is direct: compare class frequencies between a reference period and a recent window with a chi-square test or a divergence measure, and report both the absolute and the relative change. A churn rate that rises from 20% to 30% is a ten-point absolute change and a 50% relative increase; the two describe the same shift, and mixing them up misstates it.

When outcomes arrive late, score distributions can provide evidence about changing class proportions, but their sensitivity depends on the predictor. A constant or uninformative predictor may not reveal the change. Lipton, Wang, and Smola describe estimating new class proportions from unlabeled predictions and a validation confusion matrix under stable P(X|Y) and an invertible confusion matrix. Check those assumptions, estimation uncertainty, and label coverage; confirm the interpretation when representative outcomes become available.

A drop in precision or recall is a symptom, not a diagnosis. It can come from label drift, concept drift, input changes, or a pipeline defect, and the response differs for each. Labels that are observed selectively add another bias: if only flagged transactions are investigated, the measured fraud rate describes the flagged ones, not the population.

Responding

When pure label shift is a credible approximation, prior-odds adjustment or a calibration layer fitted on suitable recent labels may be sufficient. Check the assumptions and resulting decisions against current costs and capacity. For retraining, class weights w(y) = P_new(y) / P_train(y) can represent the new mix when training support covers the target classes. The size or persistence of the shift alone does not select the best remedy. Changed class definitions need a revised labeling contract; changed within-class patterns can invalidate simple prior correction. Choose calibration, new labels, or retraining based on the evidence, coverage gaps, and cost, then validate on independent data. Scheduling and validation are covered in continuous retraining.

For regression the idea carries over less cleanly. A rise in the average target is often driven by changed inputs, such as colder weather raising energy demand, and a model that uses those inputs may already account for it. A persistent bias in residuals that the inputs cannot explain is the signal to investigate.

How outcome monitoring fits into running a model over time is worked through in The Model Nobody Owned.

References

• Lipton, Wang, and Smola — Detecting and Correcting for Label Shift with Black Box Predictors (ICML 2018)
• scikit-learn — Tuning the decision threshold for class prediction