Continuous Model Management and MLOps
An illustrative retention program retrains weekly and passes its checks, yet lacks evidence of its continuing business effect.
-
-
Feedback Loop (Machine Learning)
A model can change which outcomes are observed and the outcomes themselves. Correcting the resulting bias depends on the target, coverage, and policy.
-
Label Delay
Some outcomes arrive late and some remain unobserved. Label delay requires defined outcome windows and coverage checks, not just a fixed waiting period.
-
Batch Scoring
A newly computed score can still use old inputs. Batch scoring needs a validity contract and a publication path that exposes the verified results.
-
Model Serving Architecture
A fictional checkout failure shows why stored scores, online features, fallback capacity, and recovery compatibility need explicit contracts.
-
Prediction Logging
Prediction records support replay and investigation within their coverage. Sampled features, missing outcomes, and downstream actions need separate treatment.
-
Feature Store
Two teams computing thirty-day spend two different ways is a small problem until one model trains on one definition and serves on the other.
-
Data Versioning
A query can survive while the data it once returned is lost. Data versioning records the inputs used and the limits on recovering them.
-
Model Versioning, Lineage, and Promotion
In a fictional credit-model investigation, the weights survived but training and request-time records did not. Rebuilding and replaying need different evidence.
-
Model Promotion
An alias selects a model version; deployment makes it serve traffic. Promotion needs defined acceptance criteria, authorization, and a verified recovery path.
-
AWS SageMaker
SageMaker AI connects managed training and deployment. Model artifacts, inference contracts, permissions, and workload timing determine how it is used.
-
Vertex AI
Vertex AI workflows connect training jobs, model versions, and serving resources. Integration boundaries and billing units shape platform decisions.
-
Continuous Retraining
Keeping a model current means retraining it, but a freshly trained model is only a candidate. The schedule, the triggers, and the checks it must pass first.
-
Monitoring Pipelines
A monitoring pipeline connects inputs, predictions, outcomes, and service behavior. Label delays and selective feedback shape what measurements can establish.
-
Concept Drift
The inputs look the same and the model is unchanged, but what those inputs predict has moved. Only outcomes can show it, and outcomes arrive late.
-
Data Drift
Distribution changes can alter aggregate error without changing the input-to-outcome relationship. How to interpret drift and choose a response.
-
Monitoring, Drift, and Retraining
The alarm and the harm are different things.
-
Training-Serving Skew and Feature Stores
Simulated replays separate feature leakage, staleness, and serving-input bugs. Shared definitions help, but timing and retrieval still need verification.
-
Reproducible Pipelines
Bit-identity, a stable metric, and someone else reproducing you are three different goals.
-
Prompt and Model Promotion with Evaluation Gates
A prompt or model change should reach users only with evidence. This guide builds a release gate service: versioned evaluation sets, non-inferiority thresholds, paired bootstrap intervals, slice checks, judge calibration, approvals, canary stages, and rollback, with runnable simulations.
-
Versioning Models, Data, Prompts, and Evals
A model version number says little about what the model was built from or what it needs to run. This guide builds a small lineage and registry to show which artifacts a change invalidates, what reproducibility requires, and how promotion and rollback depend on compatibility.
-
Shared AI Platform Design and Service Boundaries
A shared AI platform works only when it is clear what the platform promises and what each consuming team still owns. This guide covers consumer types, contracts, ownership, tenancy, and the runtime and evaluation services, with a runnable contract checker.
-
MLOps
Understand how MLOps connects model development, evaluation, deployment, monitoring, and recovery.