Precision (a.k.a. Positive Predictive Value, PPV)

ByJu Yeon Eum

1) Formal definition (with the confusion matrix)

  • Precision answers: “Of what I flagged as positive, how many were truly positive?”

$\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}$

  • Confusion matrix terms:
    • TP (true positive): predicted + and actually +
    • FP (false positive): predicted + but actually –
    • FN (false negative): predicted – but actually +
    • TN (true negative): predicted – and actually –

When is precision undefined?

If the model predicts no positives (TP+FP=0), precision is undefined.

  • In scikit-learn: precision_score(..., zero_division=0) (or 1) controls what to return.

2) How thresholding affects precision

  • As you raise the decision threshold, you usually keep only the most confident positives:
    • Precision ↑ (fewer false alarms),
    • Recall ↓ (you miss more actual positives).
  • There’s no single “right” threshold; choose it to satisfy business constraints (e.g., precision ≥ 0.95).

Procedure to pick a threshold for a target precision α:

  1. Sort samples by model score (desc).
  2. Sweep a cutoff downward; at each cutoff compute precision, recall.
  3. Pick the highest recall point with precision ≥ α.
  4. Lock that threshold and validate on a hold-out set.

3) Dependence on prevalence (class imbalance)

Even if your model’s TPR and FPR stay the same, precision changes with prevalence $\pi=\frac{P}{P+N}$​:

$\text{Precision}=\frac{\pi\cdot \text{TPR}}{\pi\cdot \text{TPR}+(1-\pi)\cdot \text{FPR}}$

  • If positives are rarer (smaller $\pi$), precision drops unless FPR is extremely low.
  • Practical implication: always evaluate on data with realistic class balance.

Numerical illustration (same TPR=0.60, FPR=0.002):

  • $\pi=1\%$ in 10,000 cases → $P=100$, $N=9{,}900$
    TP=60, FP≈19.8 ⇒ Precision ≈ 60 / 79.8 ≈ 0.752
  • $\pi=0.1\%$ in 10,000 cases → $P=10$, $N=9{,}990$
    TP=6, FP≈20.0 ⇒ Precision ≈ 6 / 26 ≈ 0.231

Same ROC point, very different precision due to prevalence.

4) Precision vs. related metrics

  • Recall: of all true positives, how many did we catch? $\text{TP}/(\text{TP}+\text{FN})$
  • F1-score: harmonic mean of precision and recall
    • $\text{F1}=\frac{2}{\frac{1}{\text{Precision}}+\frac{1}{\text{Recall}}}$​
  • Use F$_\beta$​ to emphasize precision ($\beta<1$) or recall ($\beta>1$).
  • Precision@k: precision measured only on the top-k ranked predictions—useful when only top results matter.
  • PR-AUC / Average Precision: summarize precision–recall trade-offs across all thresholds (great under heavy imbalance).

5) When precision matters most

  • High false-positive cost: e.g., medical alarms, fraud reviews, abuse moderation, safety alerts.
  • Limited review capacity: investigators can only look at a small number of alerts; you want those to be mostly correct.

6) Multiclass / multilabel precision

  • Per-class precision: one-vs-rest precision for each class.
  • Averaging:
    • Micro: pool all decisions, then compute one precision (favours common classes).
    • Macro: average per-class precision equally (treats classes uniformly).
    • Weighted: average per-class precision weighted by class support.

Always report which averaging you used.

7) Estimating uncertainty

Precision is a proportion over predicted positives $(n=\text{TP}+\text{FP})$.

8) Common pitfalls (and fixes)

  • Evaluated on resampled class balance → inflated/deflated precision.
    → Use a calibration/evaluation set with natural prevalence.
  • Label noise (especially in negatives) raises FP.
    → Tighten labeling, use consensus labeling or weak-label denoising.
  • Trivial high precision by predicting very few positives.
    → Also report recall (or F1) and the number of predicted positives.
  • Dataset shift (operational prevalence differs).
    → Monitor precision post-deployment; adapt thresholds by cohort.

9) Minimal worked example

Suppose a spam filter evaluated on 20,000 emails (prevalence 2% → 400 spam).

  • At a chosen threshold: TP=300, FP=50, FN=100, TN=19,550.
  • Precision $=300/(300+50)=0.857$
  • Recall $=300/(300+100)=0.75$
    Interpretation: When the filter flags spam, it’s correct 85.7% of the time, and it finds 75% of all spam.

10) Reporting checklist

  • Precision with recall (or F1) at the chosen threshold.
  • predicted positives (support) used to compute precision.
  • Class prevalence of the evaluation set.
  • If thresholded to meet a business rule (e.g., precision ≥ 0.95), state the threshold and show the precision–recall curve around it.
  • If multiclass, state the averaging scheme.

Similar Posts

Leave a Reply