Model Evaluation Reliability
A fictional claims model scores 0.97 offline and 0.71 on resolved production cases. Leakage, selection, and uncertain outcomes complicate both numbers.
-
-
Evaluation Protocol
An evaluation protocol records the task, data boundaries, decision rules, and uncertainty methods before results influence model selection.
-
Data Leakage
Leakage can distort evaluation through future features, preprocessing, or test feedback. Time and group splits help only when they match the intended task.
-
Stratified Group K-Fold
A chance-level problem scored 0.9198 when groups were split across folds and 0.5323 when they were kept together. The truth was 0.5000.
-
k-fold cross-validation
Selecting features on pure noise before the split turns a chance-level problem into 90 percent accuracy. Inside a pipeline it reads 0.53.
-
Target Variable
Defining churn as leaving within 7 days versus 365 moved the positive rate from 0.2318 to 0.9879 and AUROC from 0.7452 to 0.9619 on identical customers.
-
Categorical Encoding
On a column and a label that were both random, target encoding produced a training AUROC of 0.7950. Out-of-fold encoding gave 0.5043.
-
Upsampling
Interpolating a daily series to hourly makes the 18:00 value exactly 0.250 x today plus 0.750 x tomorrow. Forward fill stays at 1.000 x today.
-
Time-Based Splits
On a random walk, the level scores R² 0.83 under a chronological split and 0.97 when shuffled, while the change scores below zero. A high level R² alone does not show skill over a naive baseline.
-
Fine-Tuning Datasets and Evaluation Splits
A fine-tuning score is only as meaningful as the data behind it. Match data to the task, check labels, remove leakage, record rights, and size splits for the decision.
-
Time-Series Forecasting
The rows are not exchangeable and the baseline is strong.
-
Data Leakage and Train-Validation-Test Splits
Leakage compromises evaluation when unavailable information enters fitting or selection; it need not raise every observed score.
-
Case: A Predictive Service with Reliable Features
A worked, fictional prediction service built and run locally. A leaky and a point-in-time feature pipeline are compared, the service is evaluated and released as a bundle, and a feed stall and a defective release are detected and recovered.
-
Feature Pipelines and Point-in-Time Correctness
A feature value is a claim about what a system knew at prediction time. Learn the clocks, joins, parity checks, and backfill rules that keep that claim true.