Recall

Recall answers: of all the positives that were there, how many did the model find?

\(\text{Recall} = \dfrac{TP}{TP + FN}\)

A test given to 100 patients who have a disease and identifying 80 of them has a recall of 0.80. Two other fields call the same quantity by other names: medicine says sensitivity, and ROC analysis says true positive rate. They are the same formula.

What it ignores

The denominator is \(TP + FN\), which is every actual positive and nothing else. False positives are absent, and so are true negatives. Recall knows only what happened to cases that were genuinely positive; everything the model did to the negatives is invisible to it.

That gives recall an obvious failure mode. Predict positive for every case and recall is exactly 1.0, on any dataset, for any model, with no skill involved. A recall figure quoted without precision or the number of predictions made says nothing at all — “we catch 95 percent of fraud” is compatible with reviewing every transaction.

Why it does not move

Ignoring the negatives has a valuable consequence. Take one model at one threshold and change only how many negatives are in the test set:

import numpy as np

rng = np.random.default_rng(8)
"""positives holds the scores given to real positives; negatives is a large pool
   of real negatives to draw different-sized test sets from."""
positives = rng.beta(5, 2, 1000)
negatives = rng.beta(2, 5, 200000)
THRESHOLD = 0.6

caught = int(np.sum(positives >= THRESHOLD))
for n_neg in (1000, 10000, 100000):
    false_alarms = int(np.sum(negatives[:n_neg] >= THRESHOLD))
    recall = caught / positives.size
    precision = caught / (caught + false_alarms)
    accuracy = (caught + (n_neg - false_alarms)) / (positives.size + n_neg)
    print(f"negatives {n_neg:6d}  prevalence {positives.size/(positives.size+n_neg):.4f}  "
          f"recall {recall:.4f}  precision {precision:.4f}  accuracy {accuracy:.4f}")
# negatives   1000  prevalence 0.5000  recall 0.7500  precision 0.9591  accuracy 0.8590
# negatives  10000  prevalence 0.0909  recall 0.7500  precision 0.6494  accuracy 0.9405
# negatives 100000  prevalence 0.0099  recall 0.7500  precision 0.1584  accuracy 0.9581

Recall is 0.7500 in all three rows. Precision falls from 0.9591 to 0.1584. Accuracy rises from 0.8590 to 0.9581. One model, one cutoff, three metrics telling three different stories about it — and two of those stories are about the data rather than the model.

The reason is the shape of the confusion matrix. Recall is computed entirely within the positive column, so adding negatives cannot change it. Precision spans both columns and must change. Accuracy pools all four cells and drifts toward whichever class grows. Fawcett relies on exactly this when explaining why ROC curves “are insensitive to changes in class distribution”: both ROC axes, the true positive rate and the false positive rate, live within a single column each.

So recall transfers across populations in a way precision does not — under a prior-only shift, where the class-conditional score distributions stay put and only the mix changes. A recall measured on a held-out set whose positives look like production positives still holds there; a precision measured on the same set does not. Neither survives a shift that changes what the positives themselves look like, and neither figure measured on training data transfers at all. The flip side is that recall cannot warn you about anything happening on the negative side, and Saito and Rehmsmeier note that specificity — the negative-column counterpart — is itself easy to misread under heavy imbalance.

When missing a positive is the expensive error

Recall is the metric to optimize where a false negative costs far more than a false positive:

• Screening. A missed tumor is not caught again for a year; a false alarm costs a second scan. Screening tests are deliberately tuned for recall, with a confirmatory test behind them to restore precision.
• Safety monitoring. An undetected fault has consequences a spurious alert does not.
• Retrieval into a review queue. Anything the first stage drops can never be recovered downstream, so the first stage is a recall problem regardless of what the second stage does.

That last pattern is worth naming, because it is where recall is genuinely the right single objective: when a cheap high-recall filter feeds an expensive high-precision stage, the two metrics belong to different components rather than competing within one.

Raising it, and what each way costs

Improving the ranking can expand the available tradeoff. Retuning a threshold can also improve decision outcomes against costs, and need not lower empirical precision; choose it deliberately against those costs — see recalibrate thresholds. Where a single number covering both is needed, F1 is the usual choice, and \(F_\beta\) with \(\beta > 1\) weights recall more heavily when the asymmetry is known but not priced.

References

• Tom Fawcett, “An introduction to ROC analysis”, Pattern Recognition Letters, 27(8), 2006, 861–874
• Takaya Saito and Marc Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets”, PLOS ONE, 10(3), 2015