AUC (Area Under the Curve)

1) What “AUC” usually means

• AUC most often refers to AUROC: the Area Under the Receiver Operating Characteristic (ROC) Curve.
• ROC curve: plot TPR/Recall (y‑axis) vs FPR (x‑axis) as you sweep a decision threshold over your model’s scores.
• AUROC ∈ [0,1]:
  • 0.5 ≈ random ranking, 1.0 = perfect ranking, <0.5 = predict-the-opposite helps.

Intuition: AUROC is the probability that a randomly chosen positive example is scored higher than a randomly chosen negative example (ties count as 0.5).
This is equivalent to the Mann–Whitney U / Wilcoxon rank-sum interpretation.

2) How to compute AUROC (conceptually)

Two common ways:

a) Geometric (curve area).

1. Sort thresholds from high→low score.
2. At each threshold, compute (FPR, TPR).
3. Connect points; compute area via trapezoids.

b) Ranking (pairwise).

• Rank all examples by predicted score (high→low).
• For every (positive, negative) pair, add:
  • 1 if score(pos) > score(neg),
  • 0.5 if equal, 0 if lower.
• Divide by (#positives × #negatives).
• This directly measures how well the model orders positives ahead of negatives.

3) Why people like AUROC

• Threshold‑free: summarizes performance over all possible thresholds.
• Ranking‑based: insensitive to monotonic transformations of scores (e.g., logit vs probability).
• Imbalance‑resistant (partly): baseline stays 0.5 even when positives are rare (but see caveats).

4) AUROC vs. PR‑AUC (don’t confuse!)

• PR curve: Precision (y) vs Recall (x).
• PR‑AUC is often better when positives are rare because it focuses on performance on the positive class.
  • Baseline PR‑AUC = positive rate (e.g., if 5% positives, a random model has ≈0.05 PR‑AUC).
• AUROC treats TPR and FPR symmetrically; PR‑AUC emphasizes precision under class imbalance.
• Tip: report both when classes are skewed; choose the one aligned to business costs.

5) Interpreting values

• 0.90–1.00: excellent ranking
• 0.80–0.90: good
• 0.70–0.80: fair
• 0.50–0.70: weak to no skill
• Compare models using confidence intervals (e.g., DeLong’s test) or bootstrap CIs; small samples can mislead.

6) Practical caveats & pitfalls

• Imbalanced data: AUROC can look “good” even when precision at workable recall is poor; check PR‑AUC and precision/recall at target recall.
• Operating point matters: AUROC averages over thresholds you may never use. Always also report metrics at the threshold tied to your cost/benefit.
• Calibration vs ranking: High AUROC doesn’t mean predicted probabilities are well‑calibrated. Use calibration curves/Brier score for that.
• Ties & discrete scores: Many ties (e.g., small score granularity) reduce AUROC (ties count as 0.5).
• Shifted class priors: AUROC is invariant to class prior, but your optimal threshold is not.

7) Variants & extensions

• Partial AUC: area over a region of interest (e.g., low FPR zone) when certain error types are costly.
• Gini coefficient (credit scoring): Gini = 2×AUROC − 1.
• Multiclass AUROC: one‑vs‑rest (macro/micro averaging) or pairwise (one‑vs‑one) averaging schemes—be explicit which you use.
• Average Precision (AP) vs PR‑AUC: AP is a specific interpolation of the PR curve; many libraries report AP (e.g., average_precision_score)—close but not identical to trapezoidal PR‑AUC. Always state which you report.

8) Small worked example (pairwise view)

Suppose 3 positives and 3 negatives with model scores (higher is better):

• Positives: 0.92, 0.70, 0.30
• Negatives: 0.85, 0.60, 0.20

All 3×3=9 pos–neg pairs:

• 0.92 beats (0.85, 0.60, 0.20) → 3 wins
• 0.70 beats (0.60, 0.20) but not 0.85 → 2 wins
• 0.30 beats (0.20) but not (0.85, 0.60) → 1 win
  Total wins = 6 → AUROC = 6/9 = 0.667 (no ties here).

9) What to report in practice

• AUROC + PR‑AUC (especially for rare positives)
• Precision/Recall at a business‑relevant threshold (or target recall)
• Confidence intervals (DeLong or bootstrap)
• A note on calibration if probabilities drive decisions