Model Evaluation Reliability
A fictional claims model scores 0.97 offline and 0.71 on resolved production cases. Leakage, selection, and uncertain outcomes complicate both numbers.
-
-
Evaluation Protocol
An evaluation protocol records the task, data boundaries, decision rules, and uncertainty methods before results influence model selection.
-
Early Stopping
Stopping where the validation set said cost 0.0010 of test loss against the best available. Not stopping at all cost 0.0965.