Imbalanced Data

A rare positive class raises several questions: are there enough informative positive examples to learn from, which errors matter, and how will performance be measured? Resampling and class weights change training. A threshold changes decisions from an already fitted score. They address related but different parts of the problem, and neither rebalancing nor leaving the training distribution unchanged guarantees the best model.

Start with the decision and the data

If 1% of cases are positive, predicting negative for everyone achieves 99% accuracy and zero positive recall. Accuracy still measures overall correctness, but it hides that failure unless reported with class-specific results and an appropriate baseline. Precision asks what fraction of flagged cases are positive; recall asks what fraction of positives are flagged. Review positive counts as well as percentages: 1% of a million examples and 1% of a thousand provide very different amounts of minority data.

Choose an evaluation split that matches deployment. Stratification preserves class proportions in random splits, but it does not prevent leakage between related people, devices, or future and past observations. Use group or time splits where needed. Keep validation and test data representative of the target population; a balanced evaluation sample changes precision and probability metrics unless the evaluation explicitly accounts for its sampling design.

The threshold 0.5 is appropriate for calibrated deployment probabilities with equal false-positive and false-negative costs and zero cost for correct decisions. For possibly unequal positive error costs \(C_{FP}\) and \(C_{FN}\), keeping the other assumptions gives the threshold \(C_{FP}/(C_{FP}+C_{FN})\): flag when the expected false-positive cost \((1-p)C_{FP}\) is smaller than the missed-positive cost \(pC_{FN}\). A review-capacity limit instead suggests evaluating the highest-scoring cases within that budget. F1 is the harmonic mean of precision and recall, \(2PR/(P+R)\) when the denominator is nonzero. It is another possible objective, but it does not encode arbitrary costs or reward true negatives.

What training changes can do

Class weighting increases or decreases each class’s contribution to the fitting objective. Random oversampling repeats minority examples; it changes their influence but does not create independent observations. Random undersampling removes majority examples, reducing computation while potentially discarding useful boundary information. SMOTE creates interpolated minority examples. These methods can change coefficient estimates, ranking, and optimization behavior as well as the score scale. They do not expand a fixed model family’s representational capacity.

For minibatch training, minority-aware sampling can make rare examples appear more regularly. For a full-batch estimator, the issue is their contribution to the objective rather than whether the optimizer ever encounters them. Rebalancing can help or hurt depending on the model, noise, and data geometry. Compare it with an unweighted baseline and tune the operating threshold for each method under the same validation protocol.

Compare on an untouched test set

The experiment below uses a synthetic binary classification problem and five logistic-regression training strategies. The split is 40% training, 20% validation, and 40% test. Scaling is learned only from training rows, resampling touches only those rows, and F1 thresholds are selected on validation data. Test labels are used for reporting, not threshold selection. Run the blocks in order with NumPy, scikit-learn, SciPy, and imbalanced-learn installed.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (roc_auc_score, average_precision_score,
                             brier_score_loss, f1_score, precision_recall_curve)
from imblearn.over_sampling import RandomOverSampler, SMOTE
from imblearn.under_sampling import RandomUnderSampler

X, y = make_classification(n_samples=60000, n_features=20, n_informative=8,
                           weights=[0.99], flip_y=0.01, class_sep=0.9, random_state=0)
Xdev, Xtest, ydev, ytest = train_test_split(
    X, y, test_size=0.4, random_state=0, stratify=y)
Xtrain, Xval, ytrain, yval = train_test_split(
    Xdev, ydev, test_size=1/3, random_state=1, stratify=ydev)
scaler = StandardScaler().fit(Xtrain)
Ztrain, Zval, Ztest = [scaler.transform(v) for v in (Xtrain, Xval, Xtest)]

def select_f1_threshold(labels, scores):
    precision, recall, thresholds = precision_recall_curve(labels, scores)
    if len(thresholds) == 0:
        raise ValueError("Need validation scores")
    denominator = precision[:-1] + recall[:-1]
    f1 = np.divide(2 * precision[:-1] * recall[:-1], denominator,
                   out=np.zeros_like(denominator), where=denominator > 0)
    return float(thresholds[np.argmax(f1)])

strategies = [
    ("baseline", None, None),
    ("class weights", None, "balanced"),
    ("random over", RandomOverSampler(random_state=0), None),
    ("SMOTE", SMOTE(random_state=0), None),
    ("random under", RandomUnderSampler(random_state=0), None),
]
results = {}
print("positive counts", int(ytrain.sum()), int(yval.sum()), int(ytest.sum()))
print("test prevalence", round(ytest.mean(), 4))
print("strategy       ROC-AUC      AP   Brier  F1@.5 F1@val      t mean_p fit_pos")
for name, sampler, weight in strategies:
    Xfit, yfit = sampler.fit_resample(Ztrain, ytrain) if sampler else (Ztrain, ytrain)
    clf = LogisticRegression(max_iter=2000, class_weight=weight).fit(Xfit, yfit)
    pval = clf.predict_proba(Zval)[:, 1]
    ptest = clf.predict_proba(Ztest)[:, 1]
    threshold = select_f1_threshold(yval, pval)
    results[name] = (ptest, threshold)
    print(f"{name:14s} {roc_auc_score(ytest, ptest):7.4f}"
          f" {average_precision_score(ytest, ptest):7.4f}"
          f" {brier_score_loss(ytest, ptest):7.4f}"
          f" {f1_score(ytest, ptest >= 0.5):6.4f}"
          f" {f1_score(ytest, ptest >= threshold):6.4f}"
          f" {threshold:6.3f} {ptest.mean():6.4f} {yfit.mean():7.4f}")
# positive counts 369 184 369
# test prevalence 0.0154
# strategy       ROC-AUC      AP   Brier  F1@.5 F1@val      t mean_p fit_pos
# baseline        0.7750  0.4679  0.0105 0.3879 0.5396  0.210 0.0153  0.0154
# class weights   0.7814  0.4111  0.1504 0.0912 0.4611  0.822 0.3481  0.0154
# random over     0.7817  0.4076  0.1501 0.0923 0.4540  0.822 0.3475  0.5000
# SMOTE           0.7793  0.4100  0.1450 0.0941 0.4655  0.858 0.3349  0.5000
# random under    0.7818  0.3729  0.1670 0.0790 0.4226  0.897 0.3623  0.5000

AP is average precision, computed by average_precision_score; it is not a trapezoidal area under the precision–recall curve. F1@val is test F1 at the validation-selected threshold, not the largest F1 obtainable by searching the test labels. The threshold search considers distinct validation scores; its validation optimum is still subject to selection noise. If choosing among strategies or other hyperparameters, make that choice on validation or inner cross-validation before the final test comparison.

The generator starts with a nominal 1% minority and then randomly reassigns some labels, so the observed prevalence is closer to 1.5%. The fit_pos column reports the fraction of positive rows after resampling. Class weighting leaves those rows unchanged but changes their loss contributions. For binary class_weight="balanced", class c receives weight n/(2 n_c), making the two classes’ total training weight equal. A mean prediction on the original test set is not the resampled prevalence.

In this run, the baseline has AP 0.4679 and test F1 0.5396 at its validation-selected threshold. Class weighting gives AP 0.4111 and F1 0.4611, while ROC-AUC rises from 0.7750 to 0.7814. The resampled rows also have lower AP and F1 here. These metrics evaluate different aspects of the scores, so the ROC-AUC increase does not contradict the other decreases. Read ranking, probability quality, and operating-point results separately.

This comparison describes one generator, split, estimator, and set of hyperparameters. It does not establish that a small ROC-AUC difference is sampling noise, that resampling never helps, or that one method is generally superior. Shared nominal regularization settings also need care: changing the number or total weight of training examples can change the balance between fitted loss and the regularization penalty. Repeat comparisons and tune inside the training/validation procedure when making a model choice.

Probability quality is more than the mean

Calibration means that, among cases assigned a probability near 0.1, roughly 10% should be positive, and similarly at other probability levels. Brier score is mean squared probability error. It reflects both calibration and discrimination-related properties; a fifteenfold Brier increase is not a fifteenfold increase in an isolated calibration component. A mean prediction close to prevalence checks only the average. Predictions can have that same mean and still be systematically wrong for different groups of examples. Use calibration curves or other suitable calibration diagnostics on representative held-out data, with uncertainty when counts are small.

labels = np.array([0, 0, 1, 1])
for name, probabilities in [("aligned", np.array([0.1, 0.1, 0.9, 0.9])),
                            ("reversed", np.array([0.9, 0.9, 0.1, 0.1]))]:
    print(name, "mean", probabilities.mean(),
          "Brier", round(brier_score_loss(labels, probabilities), 2))
# aligned mean 0.5 Brier 0.01
# reversed mean 0.5 Brier 0.81

Both means equal the observed prevalence 0.5, yet their Brier scores are 0.01 and 0.81. This is a finite arithmetic example, not a population calibration estimate. Likewise, doubling a model’s average prediction does not imply that every predicted probability, or every probability-times-amount calculation, doubled.

Correcting a changed class prior

Suppose sampling changes prevalence from \(\pi=P(Y=1)\) to \(\pi_s\) but preserves \(P(X\mid Y)\). If q is the correct posterior for the sampled population, the target posterior p satisfies:

\[\operatorname{logit}(p)=\operatorname{logit}(q)+\operatorname{logit}(\pi)-\operatorname{logit}(\pi_s),\qquad\operatorname{logit}(u)=\log\frac{u}{1-u}.\]

This follows by separating posterior odds into a likelihood ratio and prior odds. Sampling independently within each class preserves the likelihood ratio in the population model. The correction is a constant log-odds shift, so it preserves ranking. It is not a constant multiplier on probabilities. Estimated models can deviate from these assumptions because of finite samples, model misspecification, regularization, or feature-dependent sampling.

from scipy.special import expit, logit

def prior_correct(q, target_prior, sampled_prior):
    q = np.asarray(q, dtype=float)
    if (not np.isfinite(q).all() or np.any((q <= 0) | (q >= 1))
            or not 0 < target_prior < 1 or not 0 < sampled_prior < 1):
        raise ValueError("Use probabilities and priors strictly between zero and one")
    return expit(logit(q) + logit(target_prior) - logit(sampled_prior))

original = np.array([0.01, 0.1, 0.5, 0.9])
target_prior, sampled_prior = 0.1, 0.5
sampled = expit(logit(original) + logit(sampled_prior) - logit(target_prior))
restored = prior_correct(sampled, target_prior, sampled_prior)
print("sampled posterior", np.round(sampled, 4).tolist())
print("restored posterior", np.round(restored, 4).tolist())
print("restored", np.allclose(restored, original))
# sampled posterior [0.0833, 0.5, 0.9, 0.9878]
# restored posterior [0.01, 0.1, 0.5, 0.9]
# restored True

The four numbers are hypothetical posteriors at four inputs; their average is not intended to equal the population prior. The calculation verifies the odds identity. It does not show that corrected SMOTE probabilities will be calibrated. SMOTE changes the within-class feature distribution, so a prior-only correction is not generally justified. In the ideal population weighted-log-loss problem, positive weights w1 for positive examples and w0 for negative examples multiply posterior odds by w1/w0; finite fitted classifiers need not differ only by that shift. Another option is a calibrator fitted on a separate representative validation set or with cross-fitting, followed by independent evaluation. An unweighted baseline also needs its calibration checked.

When SMOTE’s geometry is plausible

SMOTE chooses a minority example x, one of its minority neighbors \(x_n\), and creates \(x_{new}=x+u(x_n-x)\) with u between zero and one. This encourages the fitted model to treat nearby interpolations as minority examples. It works best when the chosen neighborhood and feature representation make such interpolations useful; it does not verify their labels.

Disconnected clusters do not automatically cause failure if each point’s chosen neighbors stay within its cluster. Problems arise when neighbor connections cross majority regions, bridge inappropriate clusters, or include mislabeled points. High-dimensional distance can also be uninformative, depending on feature scaling and data structure. Scale continuous features inside the training fold and inspect the induced neighborhoods. Synthetic examples add a geometric assumption, not independent information about unseen positive cases.

Ordinary SMOTE on category codes can create meaningless values, even after standardization. For mixed continuous and categorical data, SMOTENC treats categorical features separately; purely categorical data requires another approach such as SMOTEN or random duplication. Domain constraints can remain even with the appropriate sampler. A credit-default table with coded education, marital status, or repayment categories therefore needs explicit feature treatment before interpolation; treating every column as continuous would confound a comparison of sampling methods.

Minority class size must also support the neighbor calculation in each training fold; the usual five-neighbor SMOTE requires at least six minority examples there. With few positives, inspect label errors, uncertainty, and feature coverage. Smaller models, useful additional measurements, or new representative labeled cases may help. No universal class ratio determines whether resampling is appropriate.

Keep sampling inside model selection

A sampler is a fitted training operation. Running it before splitting can create synthetic validation examples related to training rows, and it also changes the evaluation distribution. The resulting score difference cannot be attributed entirely to leakage when the evaluation rows themselves differ. Fit scaling and resampling inside each training fold; validation and test rows should not be synthesized or balanced simply to match training.

For a deployment decision, compare the class-weighted or resampled model with an unweighted model at useful operating points, such as precision at a review budget or recall at an acceptable false-positive rate. Thresholds cannot repair a poor ranking, while retraining may change the ranking. Choose thresholds and calibration procedures without inspecting the final test labels. A positive class can be rare and still learnable, or relatively common but poorly measured; its percentage alone is not a diagnosis.

Exercises

1. Fold-local SMOTE. Compare logistic regression with and without SMOTE using the same validation folds. Keep scaling and sampling inside the training side of each fold.

from sklearn.model_selection import StratifiedKFold, cross_val_score
from imblearn.pipeline import Pipeline as ImbPipeline

Xcv, ycv = make_classification(n_samples=4000, n_features=30, n_informative=5,
                              weights=[0.97], flip_y=0.05, class_sep=0.5, random_state=0)
cv = StratifiedKFold(5, shuffle=True, random_state=0)
for name, sampler in [("plain", "passthrough"), ("SMOTE", SMOTE(random_state=0))]:
    pipe = ImbPipeline([("scale", StandardScaler()), ("sample", sampler),
                        ("clf", LogisticRegression(max_iter=2000))])
    aucs = cross_val_score(pipe, Xcv, ycv, cv=cv, scoring="roc_auc")
    print(name, "fold AUC", np.round(aucs, 4).tolist(), "mean", round(aucs.mean(), 4))
# plain fold AUC [0.5855, 0.692, 0.6026, 0.6731, 0.6428] mean 0.6392
# SMOTE fold AUC [0.5701, 0.6801, 0.5863, 0.6428, 0.6282] mean 0.6215

2. Can a threshold reproduce class weighting? Use the baseline and weighted scores from the main experiment. Compare rankings and decisions at their validation-selected thresholds.

from scipy.stats import spearmanr

pb, tb = results["baseline"]
pw, tw = results["class weights"]
print("rank correlation", round(spearmanr(pb, pw).statistic, 6))
print("different decisions", int(np.count_nonzero((pb >= tb) != (pw >= tw))))
for name, scores, threshold in [("baseline", pb, tb), ("weighted", pw, tw)]:
    print(name, "test F1", round(f1_score(ytest, scores >= threshold), 4),
          "validation threshold", round(threshold, 4))
# rank correlation 0.9354
# different decisions 111
# baseline test F1 0.5396 validation threshold 0.2099
# weighted test F1 0.4611 validation threshold 0.8215

3. Rarer positives and limited evidence. Compare unweighted and weighted logistic regression at nominal positive rates of 1%, 0.1%, and 0.05%. Report positive counts in both splits and ranking metrics.

for majority in (0.99, 0.999, 0.9995):
    Xr, yr = make_classification(n_samples=200000, n_features=20, n_informative=8,
                                weights=[majority], flip_y=0, class_sep=1.2, random_state=0)
    Xa, Xb, ya, yb = train_test_split(Xr, yr, test_size=0.4, random_state=0, stratify=yr)
    scale = StandardScaler().fit(Xa)
    Za, Zb = scale.transform(Xa), scale.transform(Xb)
    print("nominal positive", round(1-majority, 4),
          "train positives", int(ya.sum()), "test positives", int(yb.sum()))
    for name, weight in [("plain", None), ("weighted", "balanced")]:
        scores = LogisticRegression(max_iter=3000, class_weight=weight).fit(Za, ya).predict_proba(Zb)[:, 1]
        print(name, "AUC", round(roc_auc_score(yb, scores), 4),
              "AP", round(average_precision_score(yb, scores), 4))
# nominal positive 0.01 train positives 1200 test positives 800
# plain AUC 0.9966 AP 0.8678
# weighted AUC 0.9967 AP 0.8607
# nominal positive 0.001 train positives 120 test positives 80
# plain AUC 0.997 AP 0.6456
# weighted AUC 0.9948 AP 0.4854
# nominal positive 0.0005 train positives 59 test positives 40
# plain AUC 0.9947 AP 0.5253
# weighted AUC 0.9985 AP 0.4874

References

• Chawla et al. (2002). SMOTE: Synthetic Minority Over-sampling Technique.
• imbalanced-learn: common pitfalls and recommended practices.
• imbalanced-learn: over-sampling and mixed feature types.
• scikit-learn: probability calibration.
• scikit-learn: average precision.