Precision (a.k.a. Positive Predictive Value, PPV)

Precision answers one question: of the cases I flagged as positive, how many really were?

\(\text{Precision} = \dfrac{TP}{TP + FP}\)

Medicine calls the same quantity positive predictive value. When a filter flags 350 emails as spam and 300 of them are, its precision is 0.857 — when it fires, it is right about 86 percent of the time.

Note what the formula does not contain. True negatives are absent, so the vast majority of the data in an imbalanced problem never enters the calculation. Everything below follows from that.

It moves when the data does

Fix the model and the threshold, and change only how many negatives there are while leaving the class-conditional score distributions alone. Under that prior-only shift the true positive rate and false positive rate are both held constant. Precision still changes, because it depends on how many negatives there are to generate false alarms:

\(\text{Precision} = \dfrac{\pi \cdot \text{TPR}}{\pi \cdot \text{TPR} + (1-\pi)\cdot \text{FPR}}\)

where \(\pi\) is the prevalence of positives. The effect is not subtle.

import numpy as np

TPR, FPR = 0.60, 0.002
"""One fixed operating point on the ROC curve: 60 percent of positives caught,
   0.2 percent of negatives wrongly flagged."""

def precision_from_counts(prevalence, n):
    positives = prevalence * n
    negatives = n - positives
    tp = TPR * positives
    fp = FPR * negatives
    return tp / (tp + fp), tp, fp

def precision_from_bayes(prevalence):
    return (prevalence * TPR) / (prevalence * TPR + (1 - prevalence) * FPR)

for prevalence in (0.10, 0.01, 0.001, 0.0001):
    counted, tp, fp = precision_from_counts(prevalence, 10000)
    print(f"prevalence {prevalence:<8} TP {tp:7.1f}  FP {fp:6.1f}  "
          f"precision {counted:.4f}  (Bayes {precision_from_bayes(prevalence):.4f})")
# prevalence 0.1      TP   600.0  FP   18.0  precision 0.9709  (Bayes 0.9709)
# prevalence 0.01     TP    60.0  FP   19.8  precision 0.7519  (Bayes 0.7519)
# prevalence 0.001    TP     6.0  FP   20.0  precision 0.2309  (Bayes 0.2309)
# prevalence 0.0001   TP     0.6  FP   20.0  precision 0.0291  (Bayes 0.0291)

One model, one operating point — 60 percent of positives caught at a 0.2 percent false positive rate, which is a respectable detector — and a precision that runs from 0.9709 to 0.0291 as positives get rarer. At a prevalence of 0.01 percent, 97 percent of everything the model flags is a false alarm, and nothing about the model has changed.

The structural reason is that precision reads across a row of the confusion matrix — predicted positives, split into right and wrong — while recall and the false positive rate each read down a single column. Fawcett makes the same observation when explaining why ROC curves “are insensitive to changes in class distribution”: a metric computed within one column cannot notice how large the other column is, and a metric that spans both must.

Two practical rules follow. Evaluate on data with the prevalence you will actually see: a precision measured on a resampled 50/50 set is not the precision you will get in production. And when prevalence shifts after deployment, expect precision to move even if nothing else has.

Precision and the threshold

Raising the cutoff keeps only the most confident positives, so precision generally rises and recall falls. There is no correct threshold in the abstract, but there is a straightforward procedure when the requirement is stated as a target precision \(\alpha\):

• Sort the held-out cases by score, highest first.
• Sweep the cutoff downward, computing precision and recall at each step.
• Take the point with the highest recall whose precision is still at least \(\alpha\).
• Fix that threshold and confirm it on a second held-out set, because the first one was used to choose it.

Where costs are known in currency rather than as a precision target, the threshold can be computed instead of searched — see recalibrate thresholds.

One edge case: if the model predicts no positives at all, \(TP + FP = 0\) and precision is undefined. scikit-learn’s precision_score takes a zero_division argument to decide what to return, and the choice matters — returning 1.0 makes a model that never fires look perfect.

When precision is the metric that matters

• False positives are expensive. Medical alarms, fraud investigations, content moderation, safety alerts — each false alarm consumes attention or harms a user.
• Review capacity is fixed. If a team can look at 200 alerts a day, what matters is how many of those 200 are worth looking at.
• Only the top of the ranking is seen. Precision@k measures precision over the top \(k\) results, which is the relevant quantity for search and recommendation.

Precision alone is trivially gamed: flag one case you are certain about and precision is 1.0. It has to be reported with recall, or with the count of predicted positives, or as F1. Because precision is prevalence-dependent, Saito and Rehmsmeier argue that precision–recall plots are the more informative view on imbalanced data, where ROC plots can be “deceptive … owing to an intuitive but wrong interpretation of specificity”.

Multiclass, and reporting

Averaging	What it does
Micro	Pools every decision, then computes one precision. In single-label multiclass with one predicted label per case and all classes included, this equals accuracy, since every error is simultaneously an FP for one class and an FN for another
Macro	Averages per-class precision with equal weight, so a rare class counts as much as a common one
Weighted	Averages per-class precision weighted by support, which keeps the common classes dominant

These give different numbers on the same predictions, so the scheme has to be stated. A reported precision is complete when it comes with the recall at the same threshold, the number of predicted positives it was computed over, the prevalence of the evaluation set, and — for multiclass — the averaging. Since precision is a proportion over \(TP + FP\) cases, a Wilson or Clopper–Pearson interval is worth adding when that count is small.

References

• Tom Fawcett, “An introduction to ROC analysis”, Pattern Recognition Letters, 27(8), 2006, 861–874
• Takaya Saito and Marc Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets”, PLOS ONE, 10(3), 2015
• scikit-learn API reference, precision_score