Reweighting

Reweighting attaches a weight to each training row so that some rows count more than others. The loss becomes

\[ L = \frac{1}{N}\sum_{i=1}^{N} w_i \, \ell(f(x_i), y_i) \]

and three separate problems get handed to it: class imbalance, covariate shift, and fairness. It does something different in each, with effects that depend on assumptions and the fitted learner.

Imbalance: the threshold, not the model

The standard motivation is that on a 1% positive rate a model will predict the negative class for everything. It is worth checking what actually fails there.

Two logistic regressions are fitted to the same 80,000 rows. One is plain; the other uses class_weight="balanced", which weights each class inversely to its frequency so that fraud counts as much as legitimate traffic in total. Both are scored on the same 40,000 test rows.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from scipy.stats import spearmanr

rng = np.random.default_rng(606)
w, bias = np.array([1.3, -0.9, 0.7]), -5.4


def make(m):
    X = rng.normal(size=(m, 3))
    return X, (rng.random(m) < 1 / (1 + np.exp(-(X @ w + bias)))).astype(int)


def ece(y, p, bins=10):
    edges = np.linspace(0, 1, bins + 1)
    idx = np.clip(np.searchsorted(edges, p, "right") - 1, 0, bins - 1)
    return float(sum(abs(y[idx == b].mean() - p[idx == b].mean()) * (idx == b).mean()
                     for b in range(bins) if (idx == b).sum() > 0))


Xtr, ytr = make(80000)
Xte, yte = make(40000)
print("Fraud is %.2f%% of the training rows." % (ytr.mean() * 100))

plain = LogisticRegression(max_iter=5000).fit(Xtr, ytr)
pp = plain.predict_proba(Xte)[:, 1]
weighted = LogisticRegression(max_iter=5000, class_weight="balanced").fit(Xtr, ytr)
pw = weighted.predict_proba(Xte)[:, 1]

Fraud is 1.67% of the training rows.

For each score set, the table reports AUC, recall (the share of fraud that is flagged), precision (the share of flagged rows that are fraud) and ECE, the gap between observed rate and mean score averaged over ten score bins. The third row keeps the plain model’s scores and moves the threshold until it flags the same share of rows as the reweighted model at 0.5. The last line compares the rank order of the two score sets.

def row(label, p, thr):
    flag = p > thr
    print("%-38s %9.4f %9.4f %9.4f %9.4f"
          % (label, roc_auc_score(yte, p), yte[flag].sum() / yte.sum(),
             yte[flag].mean() if flag.any() else 0.0, ece(yte, p)))


print("%-38s %9s %9s %9s %9s" % ("", "AUC", "recall", "precision", "ECE"))
row("plain fit, decide at 0.5", pp, 0.5)
row("reweighted, decide at 0.5", pw, 0.5)
match = np.quantile(pp, 1 - (pw > 0.5).mean())
row("plain fit, same flag rate", pp, match)
print("Spearman rank correlation of the two score sets: %.6f" % spearmanr(pp, pw).statistic)

                                             AUC    recall precision       ECE
plain fit, decide at 0.5                  0.8789    0.0383    0.7353    0.0009
reweighted, decide at 0.5                 0.8789    0.8025    0.0592    0.2739
plain fit, same flag rate                 0.8789    0.8025    0.0592    0.0009
Spearman rank correlation of the two score sets: 0.999959

The plain model’s flag list at the 0.5 threshold is nearly empty, and the model is not what produced that. It flags 34 of the 40,000 test rows, 25 of them fraud, which is a recall of 0.0383 at a precision of 0.7353. Its AUC is 0.8789, so it ranks fraud well, and what left the list so short was comparing a 1.67% event against a 0.5 threshold. In this simulation the two rankings are nearly identical, with a Spearman correlation of 0.999959, and setting the plain model to flag the same share of traffic reproduces the reweighted model’s recall and precision to the digit.

What reweighting did cost is the last column: calibration error goes from 0.0009 to 0.2739, because the model was fitted to a world in which fraud is half of all traffic. A threshold change approximately reaches the displayed operating point without altering probabilities. In an ideal unrestricted posterior, constant class weighting shifts log-odds; finite samples, regularization and misspecification can also change fitted rankings. Reweight when the loss genuinely differs by class, not when the threshold is wrong.

Fairness: two cases that look identical

Reweight the under-represented group to ensure fairness is a common guardrail. Whether it does anything depends on something the sentence does not say.

The examples that follow continue in the same Python session. build draws 40,000 applicants, 20% of them in group 1, and a label from a logistic rule. In the first case both groups share the coefficient 0.9 on the first feature; in the second, group 1 has the coefficient -0.9. Group 1 rows get the weight (g == 0).sum() / (g == 1).sum(), about four, so the two groups carry equal total weight in the weighted fit. The table reports the fitted coefficients, the approval rate of group 1 at a 0.5 threshold, AUC on group 1, and the approval-rate gap between group 0 and group 1.

rng = np.random.default_rng(2468)
n = 40000


def build(group_specific):
    g = (rng.random(n) < 0.20).astype(int)
    X = rng.normal(size=(n, 2)) + np.column_stack([0.7 * (1 - g), np.zeros(n)])
    b = np.where(g == 1, -0.9 if group_specific else 0.9, 0.9)
    logit = b * X[:, 0] + 0.6 * X[:, 1] - 0.3
    return g, X, (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)


print("%-36s %-11s %8s %8s %11s %10s %15s"
      % ("case", "model", "coef x1", "coef x2", "approve g1", "AUC on g1", "gap in approve"))
for title, gs in (("the two groups follow the same rule", False),
                  ("group 1 follows a different rule", True)):
    g, X, y = build(gs)
    wts = np.where(g == 1, (g == 0).sum() / (g == 1).sum(), 1.0)
    plain = LogisticRegression(max_iter=5000).fit(X, y)
    rew = LogisticRegression(max_iter=5000).fit(X, y, sample_weight=wts)
    for label, model in (("plain", plain), ("reweighted", rew)):
        s = model.predict_proba(X)[:, 1]
        flag = s > 0.5
        print("%-36s %-11s %+8.4f %+8.4f %11.4f %10.4f %15.4f"
              % (title, label, *model.coef_[0], flag[g == 1].mean(),
                 roc_auc_score(y[g == 1], s[g == 1]),
                 flag[g == 0].mean() - flag[g == 1].mean()))

case                                 model        coef x1  coef x2  approve g1  AUC on g1  gap in approve
the two groups follow the same rule  plain        +0.8997  +0.6139      0.3920     0.7522          0.2214
the two groups follow the same rule  reweighted   +0.9000  +0.6110      0.3888     0.7522          0.2221
group 1 follows a different rule     plain        +0.5240  +0.5429      0.4322     0.4620          0.1944
group 1 follows a different rule     reweighted   +0.1053  +0.5036      0.4731     0.5913          0.0608

In the first correctly specified simulation, both groups share the conditional rule and reweighting changes the coefficients little.

In the second case the minority genuinely needs a different coefficient and is outvoted four to one. Reweighting pulls the fit toward it, AUC on that group goes from 0.4620 to 0.5913, and the approval gap falls from 0.1944 to 0.0608. In this simulation the gap narrows because the fit moves closer to the minority group’s own rule, which improves ranking within that group; a narrower gap is an improvement on this measure, not a demonstration that the model is correct for the group or fair.

Choose weights for a stated target population or loss and evaluate group outcomes. The presence of a group disparity alone does not tell whether reweighting helps. Even when the groups share a conditional rule, finite samples, regularization or misspecification can let group weights change predictions.

Covariate shift: the weights have a sample size

Under covariate shift, a correct density ratio with target support covered by the source and unchanged P(Y|X) converts a fixed function’s source expectation into its target expectation. This identity does not make an adaptively fitted model unbiased. Self-normalized ratio estimates and estimated weights can have finite-sample bias, while large weights increase variance.

Here the target shifts the first feature by shift standard deviations, so the density ratio of the shifted target to the unshifted source is exp(shift * x1 - shift**2 / 2). The weights are scaled to mean one, ess is the effective sample size \( (\sum w)^2 / \sum w^2 \), and the weighted and plain fits are compared by AUC on 20,000 target rows.

rng = np.random.default_rng(1919)
n = 6000


def make_shifted(m, shift):
    X = rng.normal(size=(m, 3)) + np.array([shift, 0.0, 0.0])
    return X, (rng.random(m) < 1 / (1 + np.exp(-(X @ [1.0, -0.7, 0.5] - 0.2)))).astype(int)


print("%9s %14s %16s %14s %12s"
      % ("shift", "largest weight", "effective rows", "weighted AUC", "plain AUC"))
Xtr, ytr = make_shifted(n, 0.0)
for shift in (0.0, 0.5, 1.0, 1.5, 2.0, 3.0):
    Xte, yte = make_shifted(20000, shift)
    logw = shift * Xtr[:, 0] - shift ** 2 / 2   # log density ratio of the shifted feature
    wt = np.exp(logw - logw.max())
    wt = wt / wt.mean()
    ess = wt.sum() ** 2 / (wt ** 2).sum()
    a_w = roc_auc_score(yte, LogisticRegression(max_iter=5000)
                        .fit(Xtr, ytr, sample_weight=wt).predict_proba(Xte)[:, 1])
    a_p = roc_auc_score(yte, LogisticRegression(max_iter=5000)
                        .fit(Xtr, ytr).predict_proba(Xte)[:, 1])
    print("%9.1f %14.1f %15.0f %14.4f %12.4f"
          % (shift, wt.max() / wt.mean(), ess, a_w, a_p))

    shift largest weight   effective rows   weighted AUC    plain AUC
      0.0            1.0            6000         0.7884       0.7884
      0.5            5.7            4688         0.7892       0.7896
      1.0           25.6            2337         0.7909       0.7917
      1.5           90.4             812         0.7912       0.7922
      2.0          255.3             228         0.7973       0.7994
      3.0         1105.8              23         0.7907       0.8042

By a shift of three the heaviest row carries a thousand times the weight of a typical one and the weight-based effective sample size of six thousand rows is about twenty-three. Weighting never beats leaving the weights alone here, and at the far end it loses, 0.7907 against 0.8042, illustrating a bias-variance and estimation trade-off.

Effective sample size is the number to put on the dashboard rather than the weights. When it falls off a cliff the correction has run out of data, and the answer is rows from the new world rather than heavier weights on the old ones. Note that reweighting also changes how much a model moves between retrainings, which is a stability question rather than an accuracy one.

References

• scikit-learn documentation, sklearn.linear_model.LogisticRegression, the class_weight and sample_weight parameters.
• Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017). Inherent trade-offs in the fair determination of risk scores. Innovations in Theoretical Computer Science (ITCS).