Recalibrate Thresholds

To recalibrate thresholds is to revise the cutoffs at which a system acts: the probability above which a classifier predicts positive, or the statistic above which a monitor raises an alert. The aim is to keep decisions aligned with their consequences when data, model behavior, capacity or costs change. In this context, “retune the threshold” is often the clearer phrase.

This differs from probability recalibration, which changes how scores map to probabilities. A threshold chooses an action from a score; a calibrator estimates what the score means. Moving a cutoff can improve decisions without fixing inaccurate probabilities, and fixing probabilities does not choose the right action for the application.

Derive a cutoff from the consequences

Let p be the estimated probability of a positive outcome. Assume that a false positive costs C_FP, a false negative costs C_FN, correct decisions cost zero, and costs are constant across cases. Predicting positive has expected cost (1 − p)C_FP. Predicting negative has expected cost pC_FN. Choose positive when the former is no larger than the latter. Rearranging gives the threshold C_FP/(C_FP + C_FN), for positive total cost.

If a false negative costs five units and a false positive costs one, the threshold is 1/6, approximately 0.1667. At p = 0.20, the expected costs are 0.80 for acting and 1.00 for not acting, so acting is cheaper under these assumptions. The lower cutoff reflects the more expensive missed positive. It does not imply that most selected cases must be negative; the selected group’s outcome rate depends on the entire distribution of scores above the cutoff.

When correct decisions also have costs, let \(C_{ij}\) mean the cost of predicting i when the outcome is j. Comparing the two expected costs gives \(p^*=(C_{10} - C_{00})/(C_{10} - C_{00}+C_{01} - C_{11})\). The usual positive-above-threshold rule assumes each kind of error costs more than its corresponding correct decision. If those inequalities do not hold, compare the expected costs directly instead of applying the formula mechanically; an action may dominate or the direction can change.

The default 0.5 is appropriate for equal false-positive and false-negative costs under the simplified assumptions. Unequal costs can justify another cutoff. If losses vary per case, such as different transaction amounts, a single global cutoff may be inadequate. Capacity limits, delayed labels and objectives such as precision at a fixed review volume introduce constraints beyond the simple two-cost calculation.

When probabilities cannot be trusted

The cost formula assumes that p estimates the relevant outcome probability. An overconfident score can preserve ranking while distorting its numerical scale. Applying a probability threshold directly to that score can select the wrong set of cases. For an illustrative transformation, q = p²/[p² + (1 − p)²] maps p = 1/6 to q = 1/26, approximately 0.0385. Keeping the numerical cutoff at 1/6 on the q scale would change decisions even though ordering is unchanged.

You can instead evaluate candidate cutoffs on representative labeled validation data and calculate the observed cost at each. This selects a decision rule empirically; it does not establish that the scores are calibrated probabilities. Separate the data used to train the model, tune the cutoff and evaluate the resulting policy, or use a validation design that preserves those boundaries. With rare positives, the apparent minimum can depend on very few events. Examine nearby cutoffs and uncertainty instead of assuming the smallest observed value is the true optimum.

A cost curve may be flat near its minimum, in which case an operationally stable cutoff can be preferable to repeatedly chasing small estimated gains. It may also have meaningful steps or a narrow useful region. Inspect the actual curve and case counts; neither a flat curve nor a fixed percentage penalty for using 0.5 is a general property of threshold selection.

What changes after probability recalibration

For a strictly increasing calibrator g, the old condition score ≥ t is equivalent to g(score) ≥ g(t). Mapping the threshold through the calibrator preserves the selected set. Keeping t unchanged on the new scale need not do so. This is a mapping argument, not a claim that every calibrator preserves every decision. Isotonic calibration can create plateaus: if formerly distinct scores become equal, a cutoff may no longer separate the same cases.

Decide whether the purpose is to preserve existing behavior or to improve the policy using new probability estimates and costs. Those are different changes. Check calibration on relevant held-out data, evaluate the proposed policy’s selected volume and outcomes, and version the calibrator and cutoff together. A new calibration fit does not automatically make its output trustworthy in every population.

Revisit the policy when its inputs change

A change in false-negative cost changes the cost-derived cutoff. A change in prevalence can also make previously estimated probabilities inaccurate, so inspect calibration and class-conditional behavior before merely shifting the cutoff. If probabilities remain valid and costs remain constant, the formula itself is unchanged, although the selected volume can change and exceed review capacity. A model replacement requires checking its score scale and decision performance afresh.

Monitoring alerts use a related decision logic, but a drift statistic is not generally an incident probability. For example, moving a population-stability-index alarm from 0.1 to 0.2 cannot be justified solely by pricing an engineer’s time. Examine historical benign variation, consequential incidents, detection delay and the response triggered by the alert. Seasonality may call for a better baseline or persistence rule rather than a higher threshold that also hides real changes.

For an operational change, state the reason, evaluation period, cost or capacity assumptions, expected selected volume, and comparison with the current rule. Observe the new policy after deployment and keep a rollback path. If the cutoff adapts automatically, bound its changes and monitor the underlying model separately; otherwise the adaptation can conceal deterioration by continually relaxing the action boundary.

References

• Charles Elkan, “The Foundations of Cost-Sensitive Learning”, Proceedings of IJCAI 2001
• scikit-learn User Guide, Probability calibration