Random Undersampling

Random undersampling discards majority-class rows at random until the classes are closer to balanced. It is the cheapest response to imbalance and the easiest to reason about, which makes it a good place to see what resampling actually does and does not cost.

Throwing away 97.5% of the majority class

import numpy as np
from imblearn.under_sampling import RandomUnderSampler
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, brier_score_loss
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=20000, n_features=8, n_informative=5,
                           weights=[0.98], flip_y=0.01, random_state=0)
xtr, xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
xu, yu = RandomUnderSampler(random_state=0).fit_resample(xtr, ytr)
print(f"training rows {len(ytr)} -> {len(yu)};  majority kept {int((yu == 0).sum())} of {int((ytr == 0).sum())}")

"""tau is the event fraction in the population, ybar the event fraction in the undersampled
   training set. The offset below is the prior correction of King and Zeng."""
tau, ybar = ytr.mean(), yu.mean()
offset = np.log((1 - tau) / tau * ybar / (1 - ybar))
print(f"event fraction: population {tau:.4f}, undersampled sample {ybar:.4f};  intercept offset {offset:.4f}")

full = LogisticRegression(max_iter=5000).fit(xtr, ytr)
under = LogisticRegression(max_iter=5000).fit(xu, yu)
corrected = 1 / (1 + np.exp(-(xte @ under.coef_[0] + under.intercept_[0] - offset)))
for name, p in (("all the data", full.predict_proba(xte)[:, 1]),
                ("undersampled", under.predict_proba(xte)[:, 1]),
                ("undersampled, prior corrected", corrected)):
    print(f"{name:<31} AUROC {roc_auc_score(yte, p):.4f}  mean predicted risk {p.mean():.4f}  Brier {brier_score_loss(yte, p):.5f}")
print(f"{'actual event rate in the test set':<31}                     {yte.mean():.4f}")
# training rows 14000 -> 688;  majority kept 344 of 13656
# event fraction: population 0.0246, undersampled sample 0.5000;  intercept offset 3.6813
# all the data                    AUROC 0.8876  mean predicted risk 0.0240  Brier 0.01231
# undersampled                    AUROC 0.8959  mean predicted risk 0.2789  Brier 0.10779
# undersampled, prior corrected   AUROC 0.8959  mean predicted risk 0.0177  Brier 0.01701
# actual event rate in the test set                     0.0247

The first surprise is in the AUROC column. Fourteen thousand training rows became 688, and discrimination did not fall — 0.8876 became 0.8959. This one draw shows no loss in test AUROC, but it does not establish that the discarded rows were unnecessary or that the decision boundary stayed the same. The prior-corrected model still has a worse Brier score than the full model, 0.01701 versus 0.01231. Repeat the draw and examine ranking, calibration, and decision behavior separately before treating majority examples as redundant.

The real cost is the next column. The true event rate is 0.0247, the full model predicts 0.0240 on average, and the undersampled model predicts 0.2789 — off by a factor of eleven. It is not a bug: the model was fitted on a sample where events were half of all rows, so its probability target is the sampled population; whether the fitted scores estimate that target accurately still needs checking.

The correction is one number

Random sampling within each class preserves its conditional feature distribution in expectation but changes the posterior probability of the outcome given the features. Under a correctly specified logistic population model, this case-control change can be represented by an intercept shift. King and Zeng give the corresponding prior correction: subtract

\[ \ln\left[\frac{1-\tau}{\tau} \cdot \frac{\bar{y}}{1-\bar{y}}\right] \]

from the fitted intercept, where \(\tau\) is the event fraction in the population and \(\bar{y}\) is the event fraction in the sample you trained on. Here that is 3.6813, and applying it pulls the average predicted risk from 0.2789 back to 0.0177 and the Brier score from 0.10779 to 0.01701. The AUROC does not move, because adding a constant to every log-odds cannot change the ordering.

Note that 0.0177 still undershoots 0.0247. The correction fixes the intercept, not the slopes, and slopes estimated from 688 rows are not the slopes of the full data. So this is a repair, not a restoration — which is itself an argument for asking whether the undersampling was needed. For models without an interpretable intercept, the same prior-odds formula can be applied directly to accurate source probabilities under the same assumptions. Held-out recalibration estimates a separate score-to-risk mapping and should be assessed on the target population.

One draw is one answer

Random undersampling makes a random choice, and a different seed keeps a different 344 rows. The model, and every number you report from it, is conditional on that draw. Where the surviving majority set is small this variance is not a rounding detail.

The usual answers are to repeat the draw and average, or to keep every majority row across an ensemble of undersampled learners — one model per subset, predictions pooled — which recovers the discarded data without ever training on the full imbalanced set. That family is where undersampling is most defensible, and it is a different proposition from a single lucky sample.

Doing it without leaking

Undersample inside each cross-validation fold, on the training part only. Balancing the whole dataset first leaves the validation part balanced as well, so every score describes a population that will never arrive — imbalanced-learn documents this as the pitfall that makes reported performance over-optimistic.

And report the number that survives the change of base rate. Accuracy on a balanced sample is meaningless; average precision and calibration on data at the natural rate are the readings that transfer.

Where it fits among the alternatives

Undersampling is the right tool when the majority class is genuinely redundant and training cost is the binding constraint — 688 rows fit where 14,000 do not. The cleaning variants replace the random choice with a targeted one: Tomek links remove majority points that sit directly against a minority point, and edited nearest neighbors removes majority points whose neighborhood disagrees with them. NearMiss and cluster-based undersampling select by distance and by structure respectively.

Oversampling or SMOTE preserves the original rows and grows the training set, while majority undersampling reduces it. Both raise the minority proportion and can inflate predicted minority probabilities; the direction of the dataset-size change is not the direction of the probability distortion.

References

• King & Zeng, Logistic Regression in Rare Events Data, Political Analysis 9(2), 2001
• imbalanced-learn User Guide, Under-sampling
• imbalanced-learn User Guide, Common pitfalls and recommended practices
• van den Goorbergh et al., The harm of class imbalance corrections for risk prediction models, JAMIA 29(9), 2022