Provisioned Throughput
Reserved throughput carries a fee commitment. Capacity units, SLA terms, overages, and model or region reassignment rules determine what it provides.
-
-
Model Fallback
A useful fallback needs tested quality, independent availability, ready capacity, and a clear policy for timeouts and partial responses.
-
Context Caching
When every request starts with the same long instructions, a service can reuse its work on that prefix. It pays off only if the prefix really is identical.
-
Model Version Pinning
An exact model version limits changes caused by moving aliases. It still needs behavior monitoring, reproducibility controls, and a deprecation plan.
-
Label Delay
Some outcomes arrive late and some remain unobserved. Label delay requires defined outcome windows and coverage checks, not just a fixed waiting period.
-
Batch Scoring
A newly computed score can still use old inputs. Batch scoring needs a validity contract and a publication path that exposes the verified results.
-
Prediction Logging
Prediction records support replay and investigation within their coverage. Sampled features, missing outcomes, and downstream actions need separate treatment.
-
Feature Store
Two teams computing thirty-day spend two different ways is a small problem until one model trains on one definition and serves on the other.
-
Data Versioning
A query can survive while the data it once returned is lost. Data versioning records the inputs used and the limits on recovering them.
-
Model Promotion
An alias selects a model version; deployment makes it serve traffic. Promotion needs defined acceptance criteria, authorization, and a verified recovery path.
-
Re-scoring
The same +0.2 boost moves promoted items in one score band up 989 places on average and items in another band 366, and it does least at the top of the page.
-
Drift Detection
A drift detector compares production with a baseline and raises a flag. Choosing the windows, thresholds, and response is where the design work lies.
-
AWS SageMaker
SageMaker AI connects managed training and deployment. Model artifacts, inference contracts, permissions, and workload timing determine how it is used.
-
Vertex AI
Vertex AI workflows connect training jobs, model versions, and serving resources. Integration boundaries and billing units shape platform decisions.
-
AWS SageMaker Endpoints
SageMaker endpoints host models for prediction. Deployment state, invocation mode, input contracts, capacity lifetime, and scaling delay shape their operation.
-
Cloud Inference with Big Payloads
Large inference inputs affect transfer, memory, and deadlines. Compression, resumable upload, and asynchronous processing solve different problems.
-
Reweighting
The reweighted and plain models rank almost identically at Spearman 0.999959. What reweighting cost was calibration: 0.0009 to 0.2739.
-
Continuous Retraining
Keeping a model current means retraining it, but a freshly trained model is only a candidate. The schedule, the triggers, and the checks it must pass first.
-
Monitoring Pipelines
A monitoring pipeline connects inputs, predictions, outcomes, and service behavior. Label delays and selective feedback shape what measurements can establish.
-
Guardrails in ML
Six independent 5% checks have about 26.5% familywise false-alarm probability per evaluation; dependence changes that rate.
-
Windows (in Time-Series)
The rolling alarm fires on three days and stays quiet on seventy-four while the latency stays 40 ms high the whole time.
-
Representation Shift
Neighbor overlap falls to 0.100 while the unbiased embedding MMD estimate remains near zero, in the nearly rotationally symmetric example with old-index/new-query embeddings.
-
Classifier Two-Sample Tests (C2STs)
A random forest scores 1.0000 in training on two samples drawn from the same distribution. Held out, it scores 0.4700.
-
Energy Distance
Record the same ages in days instead of years and the energy distance goes from 0.8126 to 296.7954. The KS reading and the p-value do not move.
-
Maximum Mean Discrepancy (MMD)
Independent samples from the same distribution have a positive expected biased squared-MMD estimate; its null expectation shrinks with sample size.
-
Categorical Drift
A PSI of 0.3967 moved the referral rate down 0.44 points. A PSI of 0.0092 moved it up 2.76. The rulebook ranks them the wrong way around.
-
Macro Shifts
Eight features, not one of them past its noise floor, and their mean correlation has gone from +0.0016 to +0.1985.
-
Off-Distribution
A batch that moved as a population with not one unusual row in it, and a batch of unusual rows whose p-value would not trip a 5% alarm.
-
KS Statistic (Kolmogorov–Smirnov Statistic)
An example reaches KS 0.4015 with AUC 0.4996: local separation and overall pairwise ranking are different properties.
-
Model Stability
In this logistic simulation, regularization and bagging change spread little; more data helps. Other learners can benefit substantially from bagging.
-
Concept Drift
The inputs look the same and the model is unchanged, but what those inputs predict has moved. Only outcomes can show it, and outcomes arrive late.
-
Data Drift
Distribution changes can alter aggregate error without changing the input-to-outcome relationship. How to interpret drift and choose a response.
-
Drift Guardrails
An intercept change is not pure label shift. The example separates the assumptions behind log-odds correction from an AUC-drop gate.
-
Dataset Shift
A new population can alter model quality, but distribution change and performance loss are different claims. Diagnose the shift before choosing a correction.
-
Compute budgets
A compute budget constrains resources, spending, and experiment choices. Account for elapsed time, memory, overhead, and the value of further evidence.
-
Inference Cost (Inference $)
Inference cost depends on workload, capacity, billing units, and accounting scope. Batching savings require checking latency and successful-request counts.
-
Label Drift
A changed class mix can alter calibrated probabilities under label-shift assumptions. Check those assumptions before choosing recalibration or retraining.
-
Covariate Drift (a.k.a. Covariate Shift)
New kinds of cases can arrive while every rule the model learned still holds. Accuracy can change anyway, because it is an average over a different mix.
-
KS shift (Kolmogorov–Smirnov shift)
At 100 rows a side, 62.5% of no-drift windows cross the 0.10 line that is supposed to mean no drift. The floor falls as 1/sqrt(n); the rubric does not.
-
PSI (Population Stability Index)
PSI sums how far each bin’s share has moved between two periods. The number is easy to compute; what counts as large depends on bins and sample size.
-
Decision Flip Rate
After a data migration, many feature values differ slightly. The decision flip rate counts only the cases where a downstream decision actually changes, which is what the consumer of the data cares about.
-
Offline-Online Parity Test
Compute the same features through the batch builder and the online store for the same requests, then compare. Differences reveal unit errors and time-selection mistakes before a release.
-
Matured-Label Cohort
When outcomes are confirmed days after a prediction, the labels available today are the quick ones. A matured-label cohort waits until a group of predictions has passed its outcome window and label delay, then scores it once.
-
Filtered Vector Search
A vector index returns the nearest neighbors, and a metadata filter such as an access rule decides which of them may be shown. Where the filter runs changes how many results come back and what can leak.
-
Continuous Batching
Continuous batching lets a language-model server change which requests share the accelerator at every generation step, instead of holding a batch fixed until its slowest request finishes.
-
Goodput
Throughput counts everything a system processes, including repeated and wasted work. Goodput counts only the part that advanced the goal, which makes it the right measure for jobs that fail, restart, and recompute.
-
GPU Fragmentation
A cluster can report plenty of idle GPUs while a job waits, because the idle devices are scattered across nodes. Packing, reservation, and defragmentation keep large blocks of free capacity available.
-
Gang Scheduling
Gang scheduling admits a group of interdependent workers only when all of them can be placed together. It prevents jobs from holding resources while waiting for each other, at the price of waiting for a large enough block of free capacity.
-
Training-Serving Skew
Understand how differences between training and serving inputs can break predictions without changing the model.
-
Experiment Tracking
Learn how experiment tracking connects runs, inputs, settings, metrics, and artifacts for meaningful comparison.