Concept Drift

Concept drift is a change in the conditional relationship between inputs and outcomes, P(Y|X): the same features can correspond to different outcome probabilities over time. Google’s Machine Learning Glossary illustrates this with the changing meaning of a label such as fuel efficient. This can degrade a deployed model, but degradation is not part of the mathematical definition. If a model always predicts class 1 and the positive probability rises from 0.6 to 0.8 for the same inputs, its accuracy improves from 0.6 to 0.8 despite concept drift. Measure effects on the metrics and segments that matter.

How it differs from the other kinds

Other examples follow the same pattern. After a recession, income predicts default less well than before. During an epidemic, symptoms that used to indicate a mild illness indicate a severe one. Fraudsters who learn which signals a model uses stop producing them, so those features stop predicting fraud. The causes are changes outside the data — economic conditions, rules, behavior, and adversaries adapting. The arrival of a new kind of customer is not, by itself, concept drift: that moves the inputs, and it becomes concept drift only if the same inputs now lead to different outcomes. The general picture, including what a moving distribution does and does not imply, is set out in data drift.

Only outcomes can show it

Because concept drift is a change in what inputs lead to, it cannot be seen in the inputs alone. It shows up when predictions are compared with outcomes, and outcomes often arrive late: a loan defaults months after approval, a churn label exists only after the renewal date. Until then, input monitoring can raise suspicion but cannot confirm this kind of drift.

A falling metric is not proof of concept drift. Accuracy, ranking quality, and calibration can change with the input mix, base rate, or pipeline defects. Check data integrity and the relevant drift assumptions; label shift and a change in P(Y|X) can coexist, so these are not mutually exclusive diagnoses. Sequential detectors such as the drift detection method (DDM) family and adaptive windowing (ADWIN) can flag changes in an error stream, but that need not identify the cause. They still require the relevant outcomes to arrive. Detector design and false alarms are covered in drift detection.

Its shapes, and the responses that fit

Concept drift can be sudden, as when a new regulation changes which applications are approved overnight; gradual, as preferences shift over months; or recurring, when an earlier relationship returns, for example under a pricing regime that is reinstated periodically. The response depends on the shape. Sudden changes may warrant retraining on data from after the change only, since older examples now teach the wrong lesson. Gradual changes suit regular retraining on a recent window. Recurring changes can justify keeping models for each regime, or features that tell the model which regime applies.

Google’s glossary recommends responding through retraining, but the available outcomes and validation process limit what an update can establish. A recent window may have incomplete labels, and a candidate still needs evaluation before promotion. A training cadence alone does not guarantee that the model tracks the changing relationship. Scheduling and validation are covered in continuous retraining, and operation over a model’s lifetime in The Model Nobody Owned.

References

• Google for Developers, Machine Learning Glossary: concept drift (Creative Commons Attribution 4.0).