Class Imbalance

Class imbalance means that some outcome classes occur much more often than others. Google’s introductory lesson uses card fraud as an example in which the positive outcome can be well below one transaction in a thousand.

Fraud and equipment failure are examples of tasks where rare outcomes can matter disproportionately. Evaluation should reflect the target population and the consequences of decisions, whether or not rebalancing helps train the model.

Three effects on evaluation

Accuracy can hide rare-class failures. If fraud is 0.1% of transactions, never flagging fraud gives 99.9% accuracy. Accuracy still measures overall correctness and corresponds to equal-cost classification error, but alone it cannot establish useful fraud detection. Report the majority-class baseline and class-specific performance.

Metrics and decision costs answer different questions. When the rare class is designated positive, precision and recall describe the reliability and coverage of positive predictions. A high ROC-AUC can coexist with weak performance at the operating threshold. Neither precision nor recall is a general error-cost function: choose the decision rule using costs and capacity constraints, then report the relevant metrics. Misclassification Cost explains that distinction; Classification Metrics and Decision Thresholds develops the measures.

The test set may hold very few positives. Ten thousand transactions at 0.1% contain about ten frauds, and any recall figure computed from ten cases is highly uncertain. The count of rare cases in the evaluation data is the number to check first.

Rebalancing can change probability interpretation

Weighting a rare class or resampling training data changes the fitting objective or distribution. It can change calibration, but it does not force every predicted probability upward. Check whether scores represent probabilities in the target population before using them in a probability-based cost formula. Probability Calibration explains how to assess and, when needed, recalibrate them on independent representative data. A correction for a changed class prior requires preserved class-conditional feature distributions and an adequate probability model; synthetic sampling or other distribution changes may violate those conditions. Choosing a threshold empirically on representative validation data is a separate option and does not inherently require calibrated probabilities.

Keep training resampling within training folds, so it cannot use evaluation observations. For evaluation, a representative untouched sample is the usual choice. Deliberate class-stratified sampling can also work if selection within classes is appropriate and target-distribution adjustments and uncertainty are accounted for. Its unadjusted precision is generally not production precision. Imbalanced Data develops the techniques and the sampling assumptions.

Four questions before trusting the result

1. What population prevalence and sampling design does the evaluation target, and are any adjustments needed to estimate production metrics?
2. How many rare cases are in the test set, and how wide is the uncertainty on the rare-class metrics?
3. Which metrics are reported, and how are costs and capacity constraints used separately to choose actions?
4. Was training rebalanced, and do checks on independent target-population data show that probability calibration needs correction?

How imbalance fits among the other ways an evaluation can mislead is worked through in Ninety-Seven on the Test Set.

References:

• Google for Developers, Machine Learning Crash Course: Imbalanced datasets (Creative Commons Attribution 4.0).