Feedback Loop (Machine Learning)
A model can change which outcomes are observed and the outcomes themselves. Correcting the resulting bias depends on the target, coverage, and policy.
-
-
Label Delay
Some outcomes arrive late and some remain unobserved. Label delay requires defined outcome windows and coverage checks, not just a fixed waiting period.
-
Batch Scoring
A newly computed score can still use old inputs. Batch scoring needs a validity contract and a publication path that exposes the verified results.
-
Prediction Logging
Prediction records support replay and investigation within their coverage. Sampled features, missing outcomes, and downstream actions need separate treatment.
-
Feature Store
Two teams computing thirty-day spend two different ways is a small problem until one model trains on one definition and serves on the other.
-
Data Versioning
A query can survive while the data it once returned is lost. Data versioning records the inputs used and the limits on recovering them.
-
Model Promotion
An alias selects a model version; deployment makes it serve traffic. Promotion needs defined acceptance criteria, authorization, and a verified recovery path.
-
Model Evaluation Reliability
A fictional claims model scores 0.97 offline and 0.71 on resolved production cases. Leakage, selection, and uncertain outcomes complicate both numbers.
-
Holdout Reuse
A test set stops being an independent final check when its results guide model selection. One look per candidate does not prevent that feedback.
-
Evaluation Protocol
An evaluation protocol records the task, data boundaries, decision rules, and uncertainty methods before results influence model selection.
-
Class Imbalance
When fraud is one case in a thousand, a model that never flags fraud is 99.9% accurate. Imbalance is a fact to evaluate around, not a defect to remove.
-
ML Problem Framing
A model can be accurate and useless if it predicts the wrong thing. Framing decides what is predicted, for which decision, and how success is judged.
-
Misclassification Cost
A missed fraud and a blocked legitimate payment have different consequences. Compare those costs when choosing the action rule.
-
Machine Learning Problem Framing
A fictional churn project shows why predicting who will leave is not enough to decide whom to call. Evaluate incremental benefit, costs, and constraints.
-
Decision Threshold
The model produces a score. Someone decides which scores lead to a call, a block, or a review, and that someone is usually not the person who trained the model.
-
Data Leakage
Leakage can distort evaluation through future features, preprocessing, or test feedback. Time and group splits help only when they match the intended task.
-
AI (Artificial Intelligence)
AI includes reasoning, search, learning, and decision-making. Rules, learned models, generative capabilities, and acting systems play different roles.
-
Machine Learning (ML)
One model scored a perfect 1.0000 on the rows it was trained on and 0.3474 on new ones. Another scored 0.9842 and 0.9822.
-
Feature Values
A feature value is what one feature holds for one example. Models need numbers, so categories are encoded, numeric ranges are scaled, and missing values need an explicit rule.
-
Jaccard index
The Jaccard index is the size of the intersection divided by the size of the union. It ignores weights, and two empty sets need an explicit rule.
-
Evaluation Set
Development, regression, and independent confirmation sets serve different decisions. Sampling and selection history determine how their scores can be used.
-
Successive Halving
Successive halving spends little on all candidates and more on survivors. The allocation can save evaluation work, but small early samples and adaptive selection still affect the result.
-
Sensitivity and Specificity
Sensitivity measures detection among actual positives, while specificity measures passes among actual negatives. A rare-failure example shows why accuracy and precision answer different questions.
-
Natural Language Inference
Natural language inference compares a premise with a hypothesis. It can check whether a passage supports a claim, but support from a passage is different from truth in the world.
-
Approximate Nearest Neighbor Search
Approximate nearest-neighbor search trades exact top-k guarantees for less work. An exact scan supplies the baseline for measuring what an approximate vector index misses.
-
Behavioral Testing
Behavioral tests check what a system should do on designed inputs. Capability-based cases and relations between outputs expose failures that an aggregate accuracy score can hide.
-
Selective Prediction
A selective predictor answers some cases and defers the rest. Coverage and selective risk must be read together, while the threshold depends on confidence and the cost of deferral.
-
Cost Matrix
Different errors have different costs. A worked queue-routing example shows how to score a system by cost and choose a decision that can differ from the most probable class.
-
Negative Control
A negative control removes the signal a pipeline is supposed to detect. Its expected score depends on the task, and its construction determines which defects it can expose.
-
MinHash
MinHash estimates set overlap from matching signature positions. Shingles define what counts as similar, and banding limits which pairs need a full comparison.
-
Dataset Datasheet
A dataset datasheet travels with the data and records the information users need before trusting or reusing it. Its value is specific answers about provenance, limits, and responsibilities.
-
Relevance Judgments and Pooling
Retrieval metrics depend on relevance judgments. Pooling makes judging feasible, but a new system can retrieve relevant documents that earlier systems never put in the pool.
-
Mean Reciprocal Rank (MRR)
MRR rewards an early first relevant result. Worked query lists show what it measures, what it ignores, and why judgments and paired uncertainty matter.
-
Feature
Learn what a model feature represents and why time boundaries, units, and training-serving consistency matter.
-
Inference
Understand inference as applying a prepared model, including preprocessing, outputs, and its distinction from learning.
-
Training
Understand how training fits a model to data and how it differs from inference, validation, and deployment.