Definition
To recalibrate thresholds means to adjust the decision cutoffs or alerting limits of a model or monitoring system so that they remain valid when data distributions or business requirements change.
- In classification models → adjust the probability threshold (e.g., default 0.5) used to decide between positive/negative.
- In monitoring / guardrails → adjust alert thresholds (e.g., acceptable drift level, anomaly rate, latency).
Why It’s Needed
- Data drift: Distribution changes mean old thresholds may no longer work.
- Business shift: Cost of false positives/negatives changes.
- Model updates: A new model calibration curve may change the optimal cutoff.
- Operational noise: Metrics fluctuate more/less than before.
Examples
1. Classification Model
- Fraud model outputs probability of fraud.
- Initially: threshold = 0.5 → flag as fraud if $p > 0.5$.
- Business decides false negatives are too costly → recalibrate threshold to 0.3.
- Now more transactions are flagged (higher recall, lower precision).
2. Monitoring Drift
- PSI drift alert threshold = 0.1.
- After observing natural seasonality, you realize 0.1 is too sensitive (too many false alarms).
- Recalibrate threshold to 0.2 to reduce noise.
3. Calibration Curves
- A model may be overconfident (predicts 0.9 probability but real outcome = 0.7).
- After recalibration (Platt scaling, isotonic regression), thresholds for decision-making must also be adjusted.
How to Recalibrate
- Empirical evaluation → test thresholds on a validation set.
- Cost-sensitive analysis → set threshold that maximizes expected business value.
- Periodic retraining → revisit thresholds when retraining or after drift.
- Dynamic thresholds → adapt automatically based on recent performance.
Summary
Recalibrate thresholds = adjust decision or alert cutoffs to reflect changes in data, model calibration, or business needs.
It ensures that both classification models and monitoring guardrails remain useful, fair, and reliable over time.