Classification Metrics and Decision Thresholds

A classifier usually produces a score before it produces a label. We can evaluate how well that score ranks cases, whether it represents a probability, and what decisions made from it cost. A strong result on one of these questions does not answer the others.

The confusion matrix and what comes out of it

Call the case we want to detect positive. A true positive (TP) is a correctly flagged positive; a false positive (FP) is a negative incorrectly flagged. A false negative (FN) is a missed positive, and a true negative (TN) is a correctly unflagged negative. The confusion matrix counts these four outcomes.

	Predicted positive	Predicted negative
Actually positive	TP = 8	FN = 2
Actually negative	FP = 4	TN = 86

In this 100-case example, precision is \(8/12\approx0.667\), recall is \(8/10=0.8\), and accuracy is \(94/100=0.94\). The positive fraction, or prevalence, is \(10/100=0.1\).

Metric	Formula	Reads
Precision	\(TP/(TP+FP)\)	of the ones I flagged, how many were real
Recall / sensitivity	\(TP/(TP+FN)\)	of the real ones, how many I caught
Specificity	\(TN/(TN+FP)\)	of the negatives, how many I left alone
F1	\(2TP/(2TP+FP+FN)\)	a symmetric combination of precision and recall
Accuracy	\((TP+TN)/n\)	how often I was right, weighted by prevalence

These metrics describe a model’s predictions at a chosen threshold on a particular evaluation set. Moving the threshold can change the predicted labels and hence the metrics; between observed scores, the predictions stay the same. Precision, recall, and F1 do not directly count true negatives, so report additional quantities when the value of correct negative decisions matters. A zero denominator also needs an explicit convention, such as reporting precision as undefined when no cases are flagged.

The fundamentals article measured a majority-class predictor scoring 92.3% accuracy on an 8%-positive problem. Report prevalence and a relevant baseline alongside accuracy so that readers can judge what the model adds.

Prevalence and the ranking metrics

The ROC curve plots the true-positive rate (recall) against the false-positive rate, \(FP/(FP+TN)\), as the threshold changes. The precision–recall curve plots precision against recall. For scores 0.9, 0.7, 0.4, 0.1 with labels 1, 0, 1, 0, a threshold of 0.8 gives recall 0.5, FPR 0, and precision 1. At 0.6 they become 0.5, 0.5, and 0.5: the added case was a false positive. ROC-AUC summarizes the ROC curve. Here we summarize the PR curve with average precision (AP), computed by average_precision_score, which weights precision by increases in recall rather than using trapezoidal interpolation. In this four-case ranking, recall increases by 0.5 at the first and third cases, where precision is 1 and \(2/3\), giving \(AP=0.5\times1+0.5\times(2/3)=5/6\). The two positives outrank three of the four positive–negative pairs, so ROC-AUC is \(3/4\). These scores can rank cases even if they are not calibrated probabilities.

import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score

rng = np.random.default_rng(0)

def make(prevalence, n=200_000):
    y = (rng.random(n) < prevalence).astype(int)
    score = rng.normal(loc=np.where(y == 1, 1.0, 0.0), scale=1.0)  # separation fixed
    return y, score

print(f"{'prevalence':>11} {'ROC-AUC':>9} {'AP':>9} {'AP baseline':>12}")
for prevalence in (0.5, 0.1, 0.01, 0.001):
    y, score = make(prevalence)
    print(f"{prevalence:11.3f} {roc_auc_score(y, score):9.4f}"
          f" {average_precision_score(y, score):9.4f} {y.mean():12.4f}")
#  prevalence   ROC-AUC    AP  AP baseline
#       0.500    0.7597    0.7542       0.5017
#       0.100    0.7583    0.2925       0.1002
#       0.010    0.7637    0.0436       0.0100
#       0.001    0.7788    0.0054       0.0008

The class-conditional score distributions stay fixed: positive scores come from a Normal distribution with mean 1 and negative scores from one with mean 0, both with standard deviation 1. Their population ROC-AUC is about 0.760. The sample estimates vary from 0.7583 to 0.7788, with fewer positives available at low prevalence. AP falls from 0.7542 to 0.0054 in these runs.

ROC-AUC measures how often a random positive outranks a random negative, with half credit for ties. With the class-conditional distributions fixed, its population value does not depend on prevalence. Precision does: at a fixed threshold it is \(\pi\,TPR/[\pi\,TPR+(1-\pi)FPR]\), where \(\pi\) is prevalence. More negatives can therefore lower precision even when TPR and FPR stay the same. Prevalence is the no-information population PR baseline; an AP calculated from a finite random ranking need not equal it exactly.

ROC-AUC summarizes pairwise ranking across the score range. AP summarizes precision across recall levels and is sensitive to prevalence. Report prevalence with AP, but do not read a multiple of that baseline as the same multiple of cases found or cost saved. If false positives or review capacity constrain the application, also measure performance at the relevant operating point.

The threshold comes from the costs

A probability score is calibrated when cases receiving a score near 0.2, for example, are positive about 20% of the time. Suppose correct decisions have zero cost, a false positive costs \(c_{FP}\), and a false negative costs \(c_{FN}\). These are fixed nonnegative costs with a positive sum, and decisions have no shared capacity constraint. For probabilities appropriate to the deployment population, predict positive when \((1-p)c_{FP} < pc_{FN}\), giving the threshold \(p^*=c_{FP}/(c_{FN}+c_{FP})\). At equality the two decisions have equal expected cost. If the score is the full conditional probability, this is the Bayes decision; calibration of a coarser score justifies the comparison for cases sharing that score.

import numpy as np

rng = np.random.default_rng(1)
n = 400_000
p_true = rng.beta(1.2, 10, n)                  # the true P(y=1 | x)
y = (rng.random(n) < p_true).astype(int)       # calibrated generating probabilities; finite samples fluctuate
print(f"positives {y.mean():.4f}")

for c_fn, c_fp in ((1, 1), (5, 1), (20, 1), (1, 5)):
    ts = np.linspace(0.005, 0.995, 199)
    cost = np.array([((y == 1) & (p_true < t)).sum() * c_fn
                     + ((y == 0) & (p_true >= t)).sum() * c_fp for t in ts])
    print(f"FN={c_fn:2d} FP={c_fp:2d}  empirical optimum {ts[cost.argmin()]:.3f}"
          f"   theory c_FP/(c_FN+c_FP) = {c_fp / (c_fn + c_fp):.3f}")
# positives 0.1072
# FN= 1 FP= 1  empirical optimum 0.505   theory c_FP/(c_FN+c_FP) = 0.500
# FN= 5 FP= 1  empirical optimum 0.170   theory c_FP/(c_FN+c_FP) = 0.167
# FN=20 FP= 1  empirical optimum 0.050   theory c_FP/(c_FN+c_FP) = 0.048
# FN= 1 FP= 5  empirical optimum 0.740   theory c_FP/(c_FN+c_FP) = 0.833

The first three empirical optima are within one grid step, 0.005, of the theoretical thresholds. Sampling variability also affects these empirical minima. In the fourth row, the largest sampled probability is 0.7398975: there are no observations at or above 0.74. Every grid threshold from 0.740 through 0.995 therefore predicts all negatives and has exactly the same observed cost. argmin returns the first minimizer, 0.740. This finite sample cannot distinguish it from 0.833.

A high ROC-AUC does not establish calibration. A strictly increasing transformation of the scores preserves every rank and tie while changing their probability interpretation. Check calibration in the deployment population before applying the probability threshold formula; an uncalibrated score can instead have a threshold selected on validation data using the desired cost.

A fixed review capacity defines another decision problem. If exactly 200 cases must be investigated and each has the same review cost and value when positive, compare how many positives appear among the top 200 scores. With accurate case probabilities, selecting the 200 largest maximizes the expected number of positives found; for a learned ranking, measure how well its top 200 actually perform. Select exactly 200 with an explicit tie-breaking rule; a cutoff equal to the 200th score can select more cases when scores tie. Precision@200 is estimated on representative evaluation data. Capacity alone does not establish whether all 200 investigations are worth their cost.

Multiclass averaging

With more than two classes, first calculate each class’s metrics by treating that class as positive and all others as negative. You can report these separately or summarize them with an average. The three common averages answer different questions.

• Macro averages the per-class scores equally (macro-F1 averages the class F1 scores, not macro-precision and macro-recall), so a class with ten examples counts as much as one with ten thousand. Use it when every class matters equally.
• Micro pools all decisions before computing, so it is dominated by the large classes; micro-precision, micro-recall, and micro-F1 equal accuracy in single-label multiclass evaluation when all classes are included.
• Weighted averages the per-class scores by support, the number of true examples in each class. Weighted recall equals accuracy in the same all-class, single-label setting; weighted precision and weighted F1 need not.

A gap between macro-F1 and micro-F1 is a reason to inspect per-class scores and support. It does not identify minority-class failure by itself. With two equally sized classes, predicting only the first class gives micro-F1 0.5 and macro-F1 1/3, even though neither class is smaller. Report the per-class table alongside the average.

The same metrics on a real imbalanced problem

The default-of-credit-card-clients dataset records 30,000 Taiwanese credit accounts and whether each defaulted the following month. Its positive rate is about 22%. We compare the model with an always-negative baseline and then inspect what different thresholds detect. The example uses the fairlearn package to fetch the dataset; the first fetch requires access to its data source. The gradient-boosted tree model is used here to produce probability estimates for comparing metrics; understanding its fitting algorithm is not needed for this example. The stratified random split evaluates held-out accounts from this dataset, not performance on a future customer population.

import numpy as np
from fairlearn.datasets import fetch_credit_card
from sklearn.model_selection import train_test_split
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score, average_precision_score

cc = fetch_credit_card(as_frame=True)
X = cc.data.astype(float).to_numpy()
y = cc.target.astype(int).to_numpy()
print(f"rows {len(y):,}   positive rate {y.mean():.4f}"
      f"   ratio 1:{(1 - y.mean()) / y.mean():.1f}")

Xtr, Xte, ytr, yte = train_test_split(
    X, y, test_size=0.3, random_state=0, stratify=y)
p = HistGradientBoostingClassifier(random_state=0).fit(Xtr, ytr).predict_proba(Xte)[:, 1]

print(f"always-predict-0 accuracy {1 - yte.mean():.4f}")
print(f"model accuracy at 0.5     {((p >= 0.5).astype(int) == yte).mean():.4f}")
print(f"ROC-AUC {roc_auc_score(yte, p):.4f}   AP {average_precision_score(yte, p):.4f}"
      f"   (AP baseline {yte.mean():.4f})")
for t in (0.2, 0.3, 0.5, 0.7):
    pr = p >= t
    print(f"  threshold {t:.1f}  flagged {pr.mean():.4f}"
          f"  precision {yte[pr].mean():.4f}  recall {pr[yte == 1].mean():.4f}")
# rows 30,000   positive rate 0.2212   ratio 1:3.5
# always-predict-0 accuracy 0.7788
# model accuracy at 0.5     0.8204
# ROC-AUC 0.7765   AP 0.5560   (AP baseline 0.2212)
#   threshold 0.2  flagged 0.3470  precision 0.4246  recall 0.6660
#   threshold 0.3  flagged 0.2212  precision 0.5369  recall 0.5369
#   threshold 0.5  flagged 0.1170  precision 0.6781  recall 0.3586
#   threshold 0.7  flagged 0.0518  precision 0.7511  recall 0.1758

Predicting nobody defaults is 77.88% accurate; the fitted model reaches 82.04%. On these 9,000 test rows, that is 375 additional correct predictions. Accuracy measures the total improvement, but does not show how it is divided between finding defaulters and avoiding false alarms.

ROC-AUC is 0.7765 and AP is 0.5560, compared with a prevalence baseline of 0.2212. The AP-to-baseline ratio is about 2.5. That describes the metric, not a 2.5-fold gain in useful decisions. The threshold table gives flag rates, precision, and recall; the corresponding counts below make the tradeoff concrete.

At 0.5 the model flags 1,053 accounts: 714 true positives and 339 false positives, with 1,277 false negatives. At 0.2 it flags 3,123: 1,326 true positives and 1,797 false positives, with 665 false negatives. Lowering the threshold prevents 612 misses but adds 1,458 false alarms. Between these two choices on this sample, 0.2 is cheaper when \(612c_{FN}>1458c_{FP}\), or \(c_{FN}/c_{FP}>2.38\) for positive FP cost. This does not establish the optimum over all thresholds. These test results illustrate the tradeoff; choose an operational threshold on validation data and assess that fixed choice on untouched test data.