Class Imbalance
When fraud is one case in a thousand, a model that never flags fraud is 99.9% accurate. Imbalance is a fact to evaluate around, not a defect to remove.
-
-
Subsampling
Asking sklearn.utils.resample for 2,000 rows returned 1,890 distinct ones. The default is replace=True, which is a bootstrap, not a subsample.
-
Class Weighting
Balanced class weights changed the Brier score from 0.01231 to 0.10726. A threshold change can match recall without reproducing the weighted model.
-
SMOTE (Synthetic Minority Over-sampling Technique)
SMOTE took the average predicted risk from 0.0240 to 0.2572 against a true rate of 0.0247, while AUROC moved by only 0.012.
-
Oversampling
Oversampling the minority four times gave coefficients identical to class_weight. The ranking barely moved; the average predicted risk more than doubled.
-
NearMiss (Distance-based Undersampling)
Average precision: no undersampling 0.6740, random 0.6586, NearMiss-1 0.6551, NearMiss-3 0.6358, NearMiss-2 0.1719. AUROC barely showed the damage.
-
Cluster-based undersampling
ClusterCentroids kept 344 majority rows, of which zero were real observations. They are k-means centroids — synthetic points, not selected samples.
-
Random Undersampling
Undersampling cut 14,000 training rows to 688. AUROC went up slightly; the average predicted risk went from 0.0240 to 0.2789 against a true rate of 0.0247.
-
One-vs-Rest (OvR) AUROC
Each OvR AUROC compares one class with its own negative mixture. Interpret the mixture and finite-sample variation before comparing class scores.
-
Macro AUROC (Macro-Averaged AUROC)
Macro AUROC weights classes equally, but OvR still depends on the negative-class mixture. OvO, uncertainty, and thresholds need separate interpretation.
-
Macro F1
Averaging then combining and combining then averaging give different answers. Which one your report contains is worth knowing.
-
Macro Precision
The same predictions can score 0.27, 0.81 or 0.94 macro precision. What changed was zero_division, not the model.
-
Stratified Shuffle Split
Five splits of 1,000 rows tested 1,000 draws — but only 679 distinct rows. The other 321 were never in a test set at all.
-
Multiclass stratified CV
A class with three samples cannot appear in five folds. scikit-learn warns instead of failing, and the warning is easy to miss.
-
Reweighting
The reweighted and plain models rank almost identically at Spearman 0.999959. What reweighting cost was calibration: 0.0009 to 0.2739.
-
Bayesian Correction
A source probability of 0.80 at a balanced prior becomes 0.3077 at a target prior of 0.10, provided the label-shift and probability assumptions hold.
-
Binary Classification
One model, one set of probabilities. Moving the cut-off from 0.10 to 0.90 took precision from 0.5271 to 0.9864 and recall from 0.8869 to 0.4306.
-
Accuracy
Two models can post identical accuracy with opposite error profiles. What the number hides, and what to report beside it.
-
Per-Class Precision
Reading the per-class table alongside support and predicted counts, and why a rare class can score 0.18 precision at 0.41 recall.
-
Multiclass Precision
Micro precision is the per-class values weighted by predicted count, which is also accuracy. Weighted uses a different weight and lands above it.
-
Upsampling
Interpolating a daily series to hourly makes the 18:00 value exactly 0.250 x today plus 0.750 x tomorrow. Forward fill stays at 1.000 x today.
-
Downsampling
Keeping every 4th sample of a 300 Hz tone turned it into a 50 Hz tone. Averaging each group of four left the same 50 Hz at a fifth of the amplitude.
-
Stratified Cross-Validation
With 12 positives in 500 rows, plain five-fold produced an empty fold in 56 of 200 shuffles. Stratification avoids that but changes split variability.
-
ROC-AUC
Two models with the same area can differ 2.5-fold in what they catch at a workable false positive rate. What to report beside the number.
-
Baseline Heuristics
Always predicting the majority class scored 0.9204 accuracy against the model’s 0.9654 — and 0.0000 F1 against 0.7389.
-
Precision (a.k.a. Positive Predictive Value, PPV)
The same model at the same operating point can score 0.97 or 0.03 on precision. What changed was the prevalence, not the model.
-
Precision–Recall AUC (PR-AUC)
The same model scores 0.79 or 0.04 depending only on how many negatives are in the test set. Why that is informative rather than a flaw.