Continuous Model Management and MLOps
An illustrative retention program retrains weekly and passes its checks, yet lacks evidence of its continuing business effect.
-
-
Drift Detection
A drift detector compares production with a baseline and raises a flag. Choosing the windows, thresholds, and response is where the design work lies.
-
Monitoring Pipelines
A monitoring pipeline connects inputs, predictions, outcomes, and service behavior. Label delays and selective feedback shape what measurements can establish.
-
Leading Indicators
At a million rows a shift of 0.02 standard deviations returns a p-value of 1e-28.
-
Concept Drift
The inputs look the same and the model is unchanged, but what those inputs predict has moved. Only outcomes can show it, and outcomes arrive late.
-
Data Drift
Distribution changes can alter aggregate error without changing the input-to-outcome relationship. How to interpret drift and choose a response.
-
Dataset Shift
A new population can alter model quality, but distribution change and performance loss are different claims. Diagnose the shift before choosing a correction.
-
Covariate Drift (a.k.a. Covariate Shift)
New kinds of cases can arrive while every rule the model learned still holds. Accuracy can change anyway, because it is an average over a different mix.
-
KS shift (Kolmogorov–Smirnov shift)
At 100 rows a side, 62.5% of no-drift windows cross the 0.10 line that is supposed to mean no drift. The floor falls as 1/sqrt(n); the rubric does not.
-
PSI (Population Stability Index)
PSI sums how far each bin’s share has moved between two periods. The number is easy to compute; what counts as large depends on bins and sample size.
-
Monitoring, Drift, and Retraining
The alarm and the harm are different things.