Oversampling

Oversampling adds minority-class rows until the classes are closer to balanced. The two families are duplication — copying rows that already exist — and synthesis, where new rows are constructed, as in SMOTE.

The duplication case is worth pinning down first, because it is not really a data operation at all.

Duplicating rows is reweighting the loss

Copying every minority row four times multiplies its contribution to the loss by four. So does telling the estimator that minority rows count four times. Uniform duplication and equivalent sample weights give the same summed data loss. They define the same optimization problem when regularization and loss normalization are treated consistently; matching fitted coefficients also depends on the solution and numerical convergence. Synthetic oversampling changes feature locations and is not this equivalence.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=20000, n_features=8, n_informative=5,
                           weights=[0.98], flip_y=0.01, random_state=0)
xtr, xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)

k = 4
"""Duplicate every minority row four times, then fit the same model with a class weight of four."""
dup = np.concatenate([np.arange(len(ytr))] + [np.flatnonzero(ytr == 1)] * (k - 1))
duplicated = LogisticRegression(max_iter=2000).fit(xtr[dup], ytr[dup])
weighted = LogisticRegression(max_iter=2000, class_weight={0: 1, 1: k}).fit(xtr, ytr)
print(f"duplication and class_weight agree to floating point: {np.allclose(duplicated.coef_, weighted.coef_, atol=1e-10)}")

plain = LogisticRegression(max_iter=2000).fit(xtr, ytr)
for name, m in (("no resampling", plain), ("oversampled x4", duplicated)):
    p = m.predict_proba(xte)[:, 1]
    print(f"{name:<15} AUROC {roc_auc_score(yte, p):.4f}   mean predicted risk {p.mean():.4f}")
print(f"actual event rate in the test set                       {yte.mean():.4f}")
# duplication and class_weight agree to floating point: True
# no resampling   AUROC 0.8876   mean predicted risk 0.0240
# oversampled x4  AUROC 0.8935   mean predicted risk 0.0578
# actual event rate in the test set                       0.0247

The coefficients agree to floating point. Nothing was learned from the copies that the weight did not already express, which is the useful way to think about this uniform duplication: it changes relative loss weights without obtaining new minority observations. Random sampling with replacement can give individual rows different multiplicities.

Three consequences follow. Under those objective and fitting conditions, class weighting gets the same result without growing the training set. The often-repeated warning that duplication “causes overfitting through exact duplicates” is at best imprecise for such models — the duplicates change the weighting, not the hypothesis space. And any concern about oversampling applies equally to the class weight you might have reached for instead.

What does change: the probabilities

Look at the last column of the run. The true event rate is 0.0247; the unresampled model’s average predicted risk is 0.0240, which is about right. After oversampling it is 0.0578 — more than double the truth, and by construction, because the model was fitted on a population where minority rows were four times as common as they really are.

Meanwhile the AUROC moves from 0.8876 to 0.8935. The ranking is almost the same; the numbers attached to it are not. That pattern is the main finding of van den Goorbergh and colleagues’ study of imbalance corrections in clinical risk models: resampling produced strongly overestimated minority probabilities without better discrimination, and the apparent gain in sensitivity was reproducible by moving the threshold on the uncorrected model.

So the question to ask before oversampling is what the output is for. If a downstream decision uses the probability — expected cost, triage, pricing, anything where 3% and 30% mean different actions — oversampling breaks it unless you recalibrate afterward. If only the ordering matters, oversampling is not buying much, and if a specific operating point is the goal, setting the threshold is the direct way to reach it.

Resample inside the fold, never before the split

imbalanced-learn documents this as its principal pitfall, and it has two distinct halves.

• The test set stops representing reality. Balancing before the split leaves the evaluation set balanced too, so every reported number describes a population that will never arrive.
• Duplicates straddle the split. A copied row can land in training while its original lands in test. The model then scores a row it has memorized — ordinary leakage, and it inflates results more the more aggressively you oversample.

The remedy is to put the sampler in a pipeline so it is refitted inside each cross-validation fold, on the training part only, and to leave the validation part at its natural rate.

When it is worth it

Oversampling earns its place where the loss cannot be reweighted directly — some implementations take no sample weights — or where a minority class is so small that a learner with a minimum-samples-per-node rule never splits on it. For gradient-boosted trees and linear models, a class weight is the simpler expression of the same intent.

And a prior question is whether imbalance is the actual problem. A rare outcome with a clear signal needs no correction; a rare outcome with little signal is not made informative by copying it.

References

• imbalanced-learn User Guide, Common pitfalls and recommended practices
• imbalanced-learn User Guide, Over-sampling
• van den Goorbergh et al., The harm of class imbalance corrections for risk prediction models, JAMIA 29(9), 2022