Cost Matrix

A cost matrix assigns a price to every combination of true outcome and system decision. It turns "the model is 90% accurate" into "the model costs this much per case", which is the number that decides whether it is worth shipping.

Why accuracy is not enough

Accuracy counts all errors equally. In practice they differ. Sending a security ticket to the billing queue may cost an hour of delay and an escalation; sending a billing ticket to the technical queue costs a few minutes of reassignment. Two systems with the same accuracy can differ several-fold in cost if they make different kinds of mistakes.

Definition

Let \(y\) be the true class, \(\hat y\) the predicted class, and \(c(y, \hat y)\) the cost of that pair, with \(c(y, y) = 0\) for correct decisions (a cost matrix can also give correct decisions a nonzero cost, such as a review fee). The expected cost per case is

\[ \bar{C} = \sum_{y}\sum_{\hat y} P(y, \hat y)\, c(y, \hat y), \]

where \(P(y, \hat y)\) is the fraction of all cases whose true class is \(y\) and whose prediction is \(\hat y\). In words: multiply each cell of the confusion matrix, as a fraction of all cases, by its price and add. The unit can be money, staff minutes, or any other common unit, and the matrix is not symmetric in general.

Worked example (hypothetical)

Three queues, with costs in minutes of staff time. Rows are the true queue, columns the predicted queue:

true \ predicted	billing	technical	security
billing	0	6	30
technical	8	0	25
security	60	45	0

Suppose a labeled sample of 120 tickets gives these counts (rows are true queues):

true \ predicted	billing	technical	security
billing	40	8	2
technical	10	30	10
security	1	2	17

Accuracy is 87/120 = 0.725. Errors cost \(8 \cdot 6 + 2 \cdot 30 = 108\) minutes for billing tickets, \(10 \cdot 8 + 10 \cdot 25 = 330\) for technical, and \(1 \cdot 60 + 2 \cdot 45 = 150\) for security, 588 in total, or 4.90 minutes per ticket. The security row is small in volume but each of its three errors costs about 50 minutes on average, so it accounts for a quarter of the total.

Using the matrix to decide, not just to score

If a model outputs probabilities \(p(y \mid x)\), the cost-minimizing decision is the one with the lowest expected cost, not the highest probability:

\[ \hat y^* = \arg\min_{\hat y} \sum_{y} p(y \mid x)\, c(y, \hat y). \]

With the matrix above and \(p = (0.50, 0.20, 0.30)\) for (billing, technical, security), the expected costs of the three predictions are 19.6, 16.5 and 20.0. The most probable class, billing, is not the cheapest decision: technical wins because it is the least damaging choice when the ticket could be security. For two classes this reduces to a threshold. With a false-positive cost of 2 and a false-negative cost of 20, predict positive when \(p > 2/(2+20) \approx 0.091\), far below 0.5.

An extra "abstain" column with cost \(h\) (the cost of handing the case to a person) makes the system answer only when its cheapest answer has expected cost below \(h\). See Selective Prediction.

Conditions and cautions

• The costs are assumptions until the owners of the process agree on them. Agreeing on them is part of the value of the exercise, and results should be checked for sensitivity: if the ranking of two systems flips when a cost changes by 30%, the ranking is not reliable.
• The decision rule needs probabilities that mean what they say. Raw model scores are often miscalibrated, and then the argmin picks the wrong class.
• Costs that depend on the case (a large customer versus a small one) need a matrix per segment or a cost per example, not one matrix.
• Class-weighted training and resampling change the training loss to reflect costs, and the decision rule above does it after training. They are related, not interchangeable.
• Cost-sensitive metrics are different from precision, recall and F1, which weight false positives and false negatives in fixed ways that may not match the real prices.