Drift Detection

Drift detection is the practice of comparing a model’s recent production data or behavior with a reference and flagging changes large enough to matter. The changes it looks for are those described in data drift: inputs that shift, such as desktop users falling from 80% to 40% of traffic (covariate drift); outcome rates that shift, such as fraud rising from 1% to 3% (label drift); and relationships that shift, such as fraud tactics changing so the same features stop predicting fraud (concept drift).

A detector answers a narrow question: is this window different from that one, by this measure, by more than this amount? Whether the difference hurts the model, and what to do about it, are separate questions. Most of the design work lies in choosing the comparison so the flag is worth acting on, and in deciding what a flag sets in motion.

What to compare

The reference is either the training data, which catches the gap between what the model learned from and what it now sees, or a recent stable production period, which catches change over time. The current window is compared against it, for example the last 24 hours against the previous four weeks. Window length is a trade-off: a short window reacts quickly but is noisy and sensitive to daily or weekly cycles; a long window is stable but slow. Comparing each day with the same weekday in earlier weeks can reduce weekly seasonality as a source of false alarms, provided holidays, campaigns, and sample composition remain comparable. A rolling reference adapts to recent behavior but can absorb gradual deterioration; a fixed reference preserves the original comparison but may keep flagging an accepted change. Record and review reference updates rather than resetting the baseline merely to silence an alert.

Several signals can be compared, and they arrive at different times. Data-quality checks, such as missing rates, out-of-range values, and new categories, are immediate. Input and prediction distributions are available as soon as requests are served. Performance measures such as loss, AUC, or precision are the most direct evidence but lag, because they need labels.

How to measure the difference

For one feature at a time, the choice depends on what a difference should mean. The two-sided Kolmogorov–Smirnov statistic is the largest gap between the two cumulative distributions: at some value, what fraction of each sample is at or below it? A gap of 0.15 means a difference of 15 percentage points at the most separated cutoff. The associated test assesses sampling evidence under its assumptions; the statistic itself does not say whether a model is harmed. The usual continuous-distribution test assumes independent samples, so repeated observations from the same customer or strong time dependence require a suitable sampling or inference design.

Wasserstein-1 distance instead measures the least average distance needed to move probability mass from one distribution to the other. It retains the feature’s units. In an illustrative example, moving equally weighted values [10, 20] to [15, 25] gives a distance of 5 units. That magnitude has a different meaning for income in dollars and latency in seconds; thresholds cannot be copied between them unchanged.

For categories, align the category vocabulary and compare proportions, retaining missing and previously unseen values rather than silently dropping them. L-infinity distance takes the largest absolute share difference: proportions [0.8, 0.2] changing to [0.4, 0.6] give 0.4. A chi-square test addresses evidence against equal category distributions, while Cramér’s V summarizes association between category and sample membership. Sparse expected counts can invalidate a convenient asymptotic test. For binned numeric data, the population stability index, Kullback–Leibler divergence, and Jensen–Shannon divergence compare bin proportions; their interpretation depends on shared bin boundaries and the treatment of empty bins. Moving bin boundaries separately for each window changes the question being measured.

Managed products expose selected measures. The Model Monitoring v2 overview lists L-infinity and Jensen–Shannon divergence for categorical distributions, and Jensen–Shannon divergence for numeric distributions. This is a version-specific feature list, not a requirement to use the same measure for every model.

For several features together, maximum mean discrepancy and energy distance compare multivariate samples. A classifier two-sample test trains a model to distinguish reference from current rows, then measures that ability on held-out rows or through a suitable cross-validation procedure. Training accuracy can reflect memorization even when the populations are unchanged. Use a chance baseline appropriate to class balance and quantify uncertainty. Weak held-out performance is not proof of equal distributions, and predictive features are clues rather than proven causes. Individual feature checks can miss changes in relationships: two binary features may each remain 50% positive while changing from always equal to always opposite. A joint comparison can expose that shift. Sequential detectors instead update a statistic as observations arrive, for example accumulating departures of prediction loss from a reference level until a boundary is crossed. Their thresholds must account for repeated looks and dependence; repeatedly applying a one-off test does not inherit the same overall false-alarm probability.

Thresholds, persistence, and false alarms

Thresholds are set by the operator. The Vertex AI documentation describes the product as alerting “when deviations exceed a specified threshold” and leaves the next step, re-evaluating or retraining the model, to the user. Widely repeated cut-offs, such as treating a population stability index above 0.2 as significant, are conventions, not standards. Their meaning changes with bin choices, sample size, and the feature’s importance, so a threshold should be checked against past data: how often it would have fired during ordinary variation, and whether it would have caught changes that did matter.

Sample size cuts both ways. With millions of rows, a statistical test can flag differences too small to affect the decision, so assess effect magnitude and model sensitivity alongside the p-value. With few rows, a real change can go unnoticed. Testing hundreds of features every day creates many chances for false alarms, and correlated features tend to alert together, so a count of alerting features overstates how much evidence there is. Importance and grouping can prioritize investigation, but they are not themselves statistical corrections. A multiple-testing procedure controls a stated error criterion under its assumptions; repeated windows and overlapping samples also need to be considered.

Requiring a signal to persist, for example over three consecutive windows, suppresses isolated excursions but adds delay relative to responding to the first flag. The delay depends on the evaluation schedule and window overlap; three correlated flags are not three independent confirmations. Whether that delay is acceptable depends on impact. An all-null feature or a unit change warrants prompt integrity investigation when it threatens the prediction contract, even before labeled outcomes establish a quality loss. A missing-value fallback may keep predictions working, so the defect alone does not prove every output is wrong.

What an alert should start

An alert should start an investigation or an evaluation, not a deployment. The first step is to rule out pipeline defects, because retraining on broken inputs bakes the defect into the next model. Next comes checking whether quality changed on the affected segments, using whatever labels or proxies are available. That evidence helps choose whether to fix the pipeline, recalibrate, adjust a threshold, train a candidate, or keep watching; incomplete labels may leave several explanations unresolved. A pipeline may train a candidate automatically when a signal persists; promoting that candidate is still a separate decision with its own checks, described in continuous retraining.

How often someone reviews drift reports should follow the model’s risk. Costly, hard-to-reverse errors call for stronger detection and response arrangements, but simply tightening every threshold can overwhelm reviewers. Choose cadence, label collection, escalation, and thresholds together, based on the consequences and the changes the monitor can actually detect.

An illustrative credit example shows the sequence. A scoring model trained on data from three years ago begins to see more applicants with irregular income. The income distribution crosses its threshold for several weeks. The team confirms that the income field is being recorded the same way, then checks default rates for the growing group once enough loans have matured. The defaults are higher than the model predicted for that group, so a candidate is trained on recent data and evaluated before any change is made.

How detection, retraining, and ownership fit together over a model’s life is worked through in The Model Nobody Owned.

References

• Google Cloud, Vertex AI Model Monitoring overview (Creative Commons Attribution 4.0) — operator-specified thresholds and the distance measures used for categorical and numeric features.
• Lopez-Paz and Oquab — Revisiting Classifier Two-Sample Tests