Class Weighting

Class weighting handles imbalance in the loss rather than in the data. Each sample’s contribution is multiplied by a weight that depends on its class:

\[ L_{weighted} = \sum_{i=1}^{N} w_{y_i} \cdot \ell(f(x_i), y_i) \]

scikit-learn’s class_weight='balanced' sets \(w_c = N / (K \cdot n_c)\), so in a binary problem a class holding a fortieth of the rows receives weight 20, while the other class receives about 0.513: a ratio of 39. Every row stays, nothing is copied and nothing is synthesized, which is usually presented as the method’s advantage over resampling.

It is an advantage — but a narrower one than it sounds, because the cost that matters is not paid in rows.

What ‘balanced’ actually buys

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score, brier_score_loss, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=20000, n_features=8, n_informative=5,
                           weights=[0.98], flip_y=0.01, random_state=0)
xtr, xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
counts = np.bincount(ytr)
print(f"training counts {counts.tolist()};  'balanced' weights "
      f"{[float(round(len(ytr) / (2 * k), 4)) for k in counts]}  (ratio {counts[0] / counts[1]:.2f})")

plain = LogisticRegression(max_iter=5000).fit(xtr, ytr)
balanced = LogisticRegression(max_iter=5000, class_weight="balanced").fit(xtr, ytr)
for name, m in (("no weighting", plain), ("class_weight='balanced'", balanced)):
    p = m.predict_proba(xte)[:, 1]
    print(f"{name:<24} AUROC {roc_auc_score(yte, p):.4f}  AP {average_precision_score(yte, p):.4f}"
          f"  Brier {brier_score_loss(yte, p):.5f}  mean risk {p.mean():.4f}  recall@0.5 {recall_score(yte, p >= 0.5):.4f}")
print(f"actual event rate in the test set {yte.mean():.4f}")

pp = plain.predict_proba(xte)[:, 1]
target = recall_score(yte, balanced.predict_proba(xte)[:, 1] >= 0.5)
th = min(np.unique(pp), key=lambda t: abs(recall_score(yte, pp >= t) - target))
print(f"no weighting, threshold moved to {th:.4f}: recall {recall_score(yte, pp >= th):.4f}")
# training counts [13656, 344];  'balanced' weights [0.5126, 20.3488]  (ratio 39.70)
# no weighting             AUROC 0.8876  AP 0.6740  Brier 0.01231  mean risk 0.0240  recall@0.5 0.4392
# class_weight='balanced'  AUROC 0.9001  AP 0.6653  Brier 0.10726  mean risk 0.2761  recall@0.5 0.8311
# actual event rate in the test set 0.0247
# no weighting, threshold moved to 0.0176: recall 0.8311

Three readings, in order of how often they are missed.

The calibration is gone. The true event rate is 0.0247 and the unweighted model averages 0.0240. Weighted, it averages 0.2761 and the Brier score is nearly nine times worse. The minority weight is about 39.7 times the majority weight. This changes the population probability target of weighted log loss; it does not guarantee calibration even to that weighted target. This is the same damage that oversampling and undersampling do — “keeping all the data” does not avoid it, because the data was never the mechanism.

Recall at 0.5 rises from 0.4392 to 0.8311. Moving the unweighted model’s threshold to 0.0176 reaches that same recall, showing that the default-threshold comparison alone does not establish a gain in ranking. Matching recall does not make the models equivalent: compare precision, selected cases and relevant costs at that matched point.

The ranking summaries change in different directions: AUROC rises from 0.8876 to 0.9001 while AP falls from 0.6740 to 0.6653. A threshold move cannot cause either change. Report the metric and operating region relevant to the decision, with uncertainty, before judging whether weighting helped.

‘balanced’ is a cost ratio you did not choose

The weights here are 0.5126 and 20.3488, a ratio of 39.70 — which is exactly the ratio of class counts. That is worth stating plainly: balanced asserts that one missed positive costs as much as forty false alarms, and it asserts this because of how the data happened to be collected, not because anyone examined the decision.

Sometimes that is roughly right. Often it is wildly off in either direction — a missed fraud case may cost a thousand reviews, or twenty. When the costs are known, pass them: class_weight={0: 1, 1: 8} encodes an eight-to-one judgment you can defend, and it is a different claim from letting the sample size decide.

Practical notes

• class_weight versus sample_weight. The first assigns one weight per class; the second assigns a weight per row, which is what you need when importance varies within a class — recent rows, high-value accounts, records with better provenance. They multiply if both are given.
• Tree ensembles have a second option. balanced_subsample recomputes the weights from each bootstrap sample rather than from the full training set, which matters when the bootstrap draws differ noticeably in composition.
• Extreme weights are a real limit. At a ratio of a thousand to one, a handful of minority rows dominate the loss and the fit follows their noise. The failure is not exotic — it is the same one that makes a tiny minority class hard to learn from at all, restated in the loss.
• Recalibrate if the output is a probability. Fit the weighted model, then map its scores back to observed frequencies on held-out data at the natural rate. Recalibration estimates a separate score-to-risk mapping. A valid analytic correction may suffice under its assumptions; otherwise assess whether held-out recalibration addresses the remaining score-level distortions.

Choosing between weighting and resampling

For anything fitted by minimizing a weighted sum — linear models, gradient boosting, neural networks — integer weights and exact row-duplication oversampling give the same summed data-loss term when normalization and penalties are matched, and the weight is the cheaper way to write it: no larger training set, no duplicate rows to straddle a cross-validation split.

Resampling earns its place where weights are unavailable, where synthesis genuinely adds plausible rows, or where discarding majority rows buys tractable training time. And the option worth keeping on the table is neither: fit the model unweighted, validate its probabilities, and set the threshold where the decision belongs.

Compare average precision as a ranking measure, then recalibrate thresholds for the operating decision.

References

• scikit-learn API reference, compute_class_weight
• scikit-learn API reference, RandomForestClassifier
• van den Goorbergh et al., The harm of class imbalance corrections for risk prediction models, JAMIA 29(9), 2022