Continuous Model Management and MLOps
An illustrative retention program retrains weekly and passes its checks, yet lacks evidence of its continuing business effect.
-
-
Holdback Group
A randomized holdback can estimate a policy’s continuing effect when assignment, follow-up, treatment history, and the outcome window support the comparison.
-
Feedback Loop (Machine Learning)
A model can change which outcomes are observed and the outcomes themselves. Correcting the resulting bias depends on the target, coverage, and policy.
-
Label Delay
Some outcomes arrive late and some remain unobserved. Label delay requires defined outcome windows and coverage checks, not just a fixed waiting period.
-
Prediction Logging
Prediction records support replay and investigation within their coverage. Sampled features, missing outcomes, and downstream actions need separate treatment.
-
Drift Detection
A drift detector compares production with a baseline and raises a flag. Choosing the windows, thresholds, and response is where the design work lies.
-
Continuous Retraining
Keeping a model current means retraining it, but a freshly trained model is only a candidate. The schedule, the triggers, and the checks it must pass first.
-
Monitoring Pipelines
A monitoring pipeline connects inputs, predictions, outcomes, and service behavior. Label delays and selective feedback shape what measurements can establish.
-
Recalibrate Thresholds
Learn how to derive a decision cutoff, tune it without leakage, and reassess it after changes to probabilities, costs, data or alert behavior.
-
Guardrails in ML
Six independent 5% checks have about 26.5% familywise false-alarm probability per evaluation; dependence changes that rate.
-
Leading Indicators
At a million rows a shift of 0.02 standard deviations returns a p-value of 1e-28.
-
Classifier Two-Sample Tests (C2STs)
A random forest scores 1.0000 in training on two samples drawn from the same distribution. Held out, it scores 0.4700.
-
Categorical Drift
A PSI of 0.3967 moved the referral rate down 0.44 points. A PSI of 0.0092 moved it up 2.76. The rulebook ranks them the wrong way around.
-
Concept Drift
The inputs look the same and the model is unchanged, but what those inputs predict has moved. Only outcomes can show it, and outcomes arrive late.
-
Data Drift
Distribution changes can alter aggregate error without changing the input-to-outcome relationship. How to interpret drift and choose a response.
-
Drift Guardrails
An intercept change is not pure label shift. The example separates the assumptions behind log-odds correction from an AUC-drop gate.
-
Dataset Shift
A new population can alter model quality, but distribution change and performance loss are different claims. Diagnose the shift before choosing a correction.
-
Label Drift
A changed class mix can alter calibrated probabilities under label-shift assumptions. Check those assumptions before choosing recalibration or retraining.
-
Covariate Drift (a.k.a. Covariate Shift)
New kinds of cases can arrive while every rule the model learned still holds. Accuracy can change anyway, because it is an average over a different mix.
-
KS shift (Kolmogorov–Smirnov shift)
At 100 rows a side, 62.5% of no-drift windows cross the 0.10 line that is supposed to mean no drift. The floor falls as 1/sqrt(n); the rubric does not.
-
PSI (Population Stability Index)
PSI sums how far each bin’s share has moved between two periods. The number is easy to compute; what counts as large depends on bins and sample size.
-
Case: A Predictive Service with Reliable Features
A worked, fictional prediction service built and run locally. A leaky and a point-in-time feature pipeline are compared, the service is evaluated and released as a bundle, and a feed stall and a defective release are detected and recovered.
-
Matured-Label Cohort
When outcomes are confirmed days after a prediction, the labels available today are the quick ones. A matured-label cohort waits until a group of predictions has passed its outcome window and label delay, then scores it once.