Partial AUC (pAUC)
A pAUC that normalizes to 0.5 can be ten times better than chance. Where the baseline actually sits, and how to standardize properly.
-
-
Micro AUC
In single-label problems the pooled positive rate is 1/K, providing a no-skill reference for micro average precision rather than a guaranteed floor.
-
Macro AUC
Three classes with similar ROC-AUC score 0.96, 0.59 and 0.23 on average precision. Prevalence changes how these metrics should be read.
-
Sensitivity in Feature Engineering
Five arbitrary re-numberings of the same six categories gave logistic regression 0.8799, 0.3775, 0.6220, 0.6220 and 0.6232. The forest stayed at 0.8799.
-
Categorical Encoding
On a column and a label that were both random, target encoding produced a training AUROC of 0.7950. Out-of-fold encoding gave 0.5043.
-
Normalize (in Feature Engineering)
With fold-local scaling, k-nearest neighbors rose from 0.6215 to 0.7175. Logistic regression was 0.7282 in both raw and standardized form; forest results were nearly unchanged.
-
Accuracy
Two models can post identical accuracy with opposite error profiles. What the number hides, and what to report beside it.
-
Counterfactual Explanations
Four valid counterfactuals for one denied applicant: earn $36,528 more, work 12 more years, hold minus 3.6 accounts, or turn 108.
-
LIME
At 100 perturbations the top feature flipped between seeds. At a kernel width of 0.25 the surrogate’s R-squared was 0.0239 and it still ranked features.
-
SHAP (SHapley Additive exPlanations)
Duplicate columns split credit equally in this symmetric example. Conditional and interventional SHAP can allocate dependent-feature credit differently.
-
Post-hoc Explainability
In a synthetic example, a model never shown a protected attribute selected one group at 0.7823 and the other at 0.2502. The explanation named a postcode variable and was correct.
-
Decision Trees
Five bootstrap resamples produced trees whose root thresholds ran from -0.4341 to +0.1348 and which agreed with each other on 0.9125 to 0.9305 of rows.
-
Linear Models
The same logistic regression scored 0.6973 on two raw features and 0.9963 on their degree-2 polynomials, recovering a circular boundary.
-
Full Annotation
Leaving 80% of training positives unlabeled reduced recall from 0.6341 to 0.0031 in this experiment. Evaluation-label omissions have different effects.
-
Weak Supervision
Oracle source weights achieved 0.8088 accuracy at 93.7% coverage. Coverage also depends on how ties and abstentions are resolved.
-
Fairness Guardrails
None of seven tested thresholds passed all three guardrails. That finite search does not prove impossibility; a low-recall counterexample passes them.
-
Weighted Averaging
Weighted precision 0.8071, weighted recall 0.4600, weighted F1 0.3670 — below both, which scikit-learn’s own documentation warns about.
-
Label Noise
Random label noise in training barely moved accuracy, 0.7732 to 0.7681. The same noise in the test set took it to 0.5595, and AUROC to 0.5743.
-
Evaluation Set
Development, regression, and independent confirmation sets serve different decisions. Sampling and selection history determine how their scores can be used.
-
Per-Class Precision
Reading the per-class table alongside support and predicted counts, and why a rare class can score 0.18 precision at 0.41 recall.
-
Multiclass Precision
Micro precision is the per-class values weighted by predicted count, which is also accuracy. Weighted uses a different weight and lands above it.
-
Multilabel Precision
The identity that collapses micro precision, recall and F1 into one number does not survive samples carrying several labels.
-
Harmonic Mean
Precision 0.99 with recall 0.02 gives an arithmetic mean of 0.5050 and a harmonic mean of 0.0392. The large number is all but ignored.
-
F1-score
Why the harmonic mean rather than the average, what F1 cannot see, and when a single balanced number is the wrong thing to want.
-
Model Score
A raw ranking value, or a performance metric. The same model can score 0.95 in one sense while being no better than a constant rule.
-
Average Precision (AP)
Three different numbers go by the name PR-AUC. Which one average_precision_score returns, and why the difference is not cosmetic.
-
Upsampling
Interpolating a daily series to hourly makes the 18:00 value exactly 0.250 x today plus 0.750 x tomorrow. Forward fill stays at 1.000 x today.
-
Downsampling
Keeping every 4th sample of a 300 Hz tone turned it into a 50 Hz tone. Averaging each group of four left the same 50 Hz at a fifth of the amplitude.
-
Micro Averaging
Accuracy, micro precision, micro recall and micro F1 all read 0.8000 on the same single-label data. Four columns, one number.
-
Macro Averaging
The same predictions gave a macro precision of 0.5159 or 0.8493 depending on one keyword argument — and two macro F1 definitions gave 0.5152 and 0.5170.
-
AUC (Area Under the Curve)
The ranking view and the area view give the same value. Where the term is ambiguous, how ties are counted, and what AP is not.
-
Fairness parity
Equal error rates can coexist with unequal selection rates and precision. A worked example explains the denominators, trade-offs and limits of parity claims.
-
Interleaving Tests
Instead of splitting users between two rankers, interleaving shows each request one blended list. The credit rule turns clicks into a preference, and some rules are biased.
-
Time-Based Splits
On a random walk, the level scores R² 0.83 under a chronological split and 0.97 when shuffled, while the change scores below zero. A high level R² alone does not show skill over a naive baseline.
-
Stratified Cross-Validation
With 12 positives in 500 rows, plain five-fold produced an empty fold in 56 of 200 shuffles. Stratification avoids that but changes split variability.
-
Selection Rate
Group B was hired at a higher rate in both roles — 0.6600 and 0.0800 — and its combined impact ratio came out at 0.2629.
-
Novelty (in Recommender Systems)
A novel recommendation is one the user did not know about. Offline, novelty is usually approximated from popularity, which is a proxy and not a measurement of what a user knows.
-
Diversity (in Recommender Systems)
Diversity is the opposite of similarity, and what counts as similar is a choice. It can be measured inside one list, between users’ lists, or across the whole catalog.
-
Coverage
Coverage is an umbrella for breadth: how much of the catalog gets recommended and how many users get recommendations at all. By itself it says nothing about relevance.
-
Hit Rate
Hit rate asks whether a relevant item appears in the top K at all. It ignores position inside the list and depends on how the candidates were chosen.
-
NDCG (Normalized Discounted Cumulative Gain)
NDCG rescales DCG by the ideal ordering’s DCG. A score is only comparable when the gain convention, cutoff, candidate set, and empty-query rule are the same.
-
Mean Average Precision (MAP)
Three metrics share the name AP. This is the retrieval one, where the denominator is every relevant item rather than every retrieved one.
-
Expected Calibration Error (ECE)
One number for how far predicted probabilities sit from observed frequencies — biased upward, binning-dependent, and defined two different ways.
-
Reliability Curves
The diagnostic that shows where a model’s probabilities go wrong rather than how much. Including how much wobble to expect from a model that is right.
-
Log Loss
The negative log-likelihood, used as a metric. Strictly proper, unbounded, and sensitive enough to a single confident error to need watching.
-
Brier Score
Mean squared error for probabilities. It rewards calibration and discrimination at once, which is why it needs a baseline before it means anything.
-
Calibration quality (Model Calibration)
Learn what a calibrated probability means, how calibration differs from ranking, and how to diagnose and improve it without confusing scores with decisions.
-
Recall
Also called sensitivity and the true positive rate. Why it stays fixed while precision and accuracy move in opposite directions.
-
ROC-AUC
Two models with the same area can differ 2.5-fold in what they catch at a workable false positive rate. What to report beside the number.
-
Baseline Heuristics
Always predicting the majority class scored 0.9204 accuracy against the model’s 0.9654 — and 0.0000 F1 against 0.7389.