Misclassification Cost

Misclassification cost is what each kind of prediction error costs the organization that acts on the prediction. In a two-class problem there are four outcomes — correctly flagged, correctly passed, wrongly flagged, wrongly passed — and they almost never have equal consequences. A missed fraudulent payment loses its amount; a blocked legitimate payment loses a sale and some goodwill; an investigated case costs analyst time whether or not it turns out to be fraud.

A default probability threshold of 0.5 corresponds to equal error costs when probabilities are appropriate, correct decisions have equal zero cost, and there are no additional constraints. This describes a decision rule, not a universal assumption of the learning algorithm. The scikit-learn guide explains why an application may need a different threshold.

Writing the costs down

A cost matrix records the consequence of each outcome in units the business uses. A fictional fraud screen, per transaction:

Put all entries on a common cost basis and time window. For action \(a\), outcome \(y\), and fraud probability \(p\), compare \(L(a,p)=pC(a,1)+(1-p)C(a,0)\). Choose the lower expected cost among feasible actions; review does not have to prevent every loss. With zero costs for correct classifications, fixed positive error costs, and no other constraints, this reduces to the familiar threshold \(C_{FP}/(C_{FP}+C_{FN})\), with either action optimal at equality. Classification Metrics and Decision Thresholds derives that special case. Nonzero review costs belong in the full comparison above; variable amounts may require case-specific rules. Decision threshold explains how to operationalize them.

Why accuracy misleads when costs differ

Accuracy counts every error as one. When fraud is one transaction in a thousand, a model that passes everything is 99.9% accurate and prevents nothing. When a false negative costs a hundred times a false positive, a model with lower accuracy and more false alarms can be far more valuable. Metrics emphasize different aspects of prediction. AUC, for example, measures positive–negative pair ordering without specifying the cost of acting at one operating point. Evaluate the business consequences separately.

The same asymmetry appears in numeric predictions. Under-forecasting stock loses sales; over-forecasting ties up cash and creates waste; the two rarely cost the same per unit. The choice between error measures, and what each implicitly rewards, is discussed in Regression and Ranking Metrics.

Costs that are easy to omit

• Ignoring the value of correct predictions. A correctly flagged case still costs the review; a correctly retained customer may have cost a discount they did not need.
• Leaving out invisible costs. Customer friction, regulatory exposure, and staff time are real even when no ledger records them.
• Using averages where amounts vary. A missed fraud on a large transaction is not the average miss; when costs scale with the case, the matrix should too.
• Forgetting the system’s own cost. The model has to beat the current approach by more than it costs to run. Google’s framing course includes engineering effort and compute in the decision about further improvement.

Rough cost estimates justify checking the default, not automatically rejecting it. Suppose false negatives cost 10 units, false positives cost 1, correct decisions cost zero, and calibrated probabilities are only 0.01 and 0.99. Thresholds 0.5 and 1/11 make identical decisions and both incur expected total cost 0.11 across those two cases. Compare costs and selections over plausible estimates rather than treating a changed threshold number as an improvement. ML problem framing connects costs to the task; Predicting the Wrong Thing Well develops the decision as a whole.

References

• scikit-learn User Guide, Tuning the decision threshold for class prediction — the limits of the default rule and the diagnostic example.
• Google for Developers, Framing an ML problem (Creative Commons Attribution 4.0) — weighing improvement against engineering and compute cost.