Macro Shifts

A macro shift is a change in the world large enough to move many things at once — a rate rise, a pandemic, a policy change. In a monitoring context the useful definition is narrower and more testable: a macro shift is a common cause sitting underneath a set of features that used to move independently.

That shape defeats three habits that usually follow a shift: watching each feature on its own, recalibrating the model, and retraining once enough post-shift data has accumulated.

It hides from a per-feature dashboard

A common factor does not have to move any single column very far. It only has to move all of them a little, together.

The experiment draws eight independent standard-normal features before the shift. After it, each feature keeps unit variance but gets a shared component: shock is one value per row, mixed into all eight columns with weight 0.45. The per-feature monitor is a two-sample KS statistic for each column, compared with its 5% critical value for two samples of 900 rows, 1.36 * sqrt(2 / n). The code blocks in this entry run in order in one session, so later blocks reuse these imports and functions.

import numpy as np


def ks_stat(a, b):
    a, b = np.sort(a), np.sort(b)
    g = np.concatenate([a, b])
    return float(np.max(np.abs(np.searchsorted(a, g, "right") / a.size
                               - np.searchsorted(b, g, "right") / b.size)))


rng = np.random.default_rng(3141)
n, d = 900, 8
floor = 1.36 * np.sqrt(2 / n)

before = rng.normal(size=(n, d))
shock = rng.normal(size=(n, 1))
after = rng.normal(size=(n, d)) * np.sqrt(1 - 0.45 ** 2) + 0.45 * shock

ks = [ks_stat(before[:, j], after[:, j]) for j in range(d)]
print("Per-feature KS, against a 5%% floor of %.4f:" % floor)
print("   %s" % "  ".join("%.4f" % v for v in ks))
print("   worst %.4f, which is %.2f times the floor" % (max(ks), max(ks) / floor))
print("Mean off-diagonal correlation   before %+.4f    after %+.4f"
      % (np.mean(np.corrcoef(before.T)[np.triu_indices(d, 1)]),
         np.mean(np.corrcoef(after.T)[np.triu_indices(d, 1)])))

Per-feature KS, against a 5% floor of 0.0641:
   0.0489  0.0356  0.0456  0.0222  0.0444  0.0378  0.0378  0.0533
   worst 0.0533, which is 0.83 times the floor
Mean off-diagonal correlation   before +0.0016    after +0.1985

Not one of the eight features crosses its own noise floor — the worst reads 0.83 of it — and the mean pairwise correlation has gone from +0.0016 to +0.1985. Every gauge is green and the eight of them have started moving as one body.

A joint test reads all eight columns at once. It is an MMD permutation test: sq computes pairwise squared distances, the Gaussian kernel uses the median squared distance as its bandwidth, and mmd_p shuffles which rows count as the first sample 300 times, reporting the share of shuffles whose unbiased MMD estimate is at least the observed one.

def sq(Z):
    s = np.einsum("ij,ij->i", Z, Z)
    return np.maximum(s[:, None] + s[None, :] - 2 * Z @ Z.T, 0.0)


def mmd_p(X, Y, reps=300):
    Z = np.vstack([X, Y])
    d2 = sq(Z)
    K = np.exp(-d2 / (2 * np.median(d2[np.triu_indices(len(Z), 1)])))
    m = len(X)

    def stat(i):
        a, b = i[:m], i[m:]
        return ((K[np.ix_(a, a)].sum() - m) / (m * (m - 1))
                + (K[np.ix_(b, b)].sum() - len(b)) / (len(b) * (len(b) - 1))
                - 2 * K[np.ix_(a, b)].mean())
    idx = np.arange(len(Z))
    s0 = stat(idx)
    rng, hits = np.random.default_rng(5), 0
    for _ in range(reps):
        rng.shuffle(idx)
        hits += stat(idx) >= s0
    return (hits + 1) / (reps + 1)


print("Joint MMD permutation p         %.4f" % mmd_p(before, after))

Joint MMD permutation p         0.0033

The joint test rejects at p = 0.0033 on the same rows that no per-feature gauge flagged. This is the same blind spot KS shift describes, where a change in the correlation between features passes every per-feature gauge. Here the change comes from a common factor, and the fix is the same: a test that reads the features jointly.

Calibration repair and remaining ranking loss

When a model looks wrong after a shift, recalibration is the first thing reached for, because it is fast and needs few labels. It is worth being precise about what it can and cannot do. A logistic model is fitted once on 30,000 rows. Two shifted worlds are then generated: in one every input moves by 0.7 while the coefficients stay the same, and in the other the coefficients themselves change. ece is the expected calibration error over ten equal-width probability bins, and an isotonic calibrator is fitted on the first half of each shifted batch and evaluated on the second.

from sklearn.linear_model import LogisticRegression
from sklearn.isotonic import IsotonicRegression
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(808)
w_before = np.array([1.2, -0.8, 0.5])


def make(n, w, mu):
    X = rng.normal(size=(n, 3)) + mu
    p = 1 / (1 + np.exp(-(X @ w - 0.4)))
    return X, (rng.random(n) < p).astype(int)


def ece(y, p, bins=10):
    edges = np.linspace(0, 1, bins + 1)
    idx = np.clip(np.searchsorted(edges, p, "right") - 1, 0, bins - 1)
    return float(sum(abs(y[idx == b].mean() - p[idx == b].mean()) * (idx == b).mean()
                     for b in range(bins) if (idx == b).sum() > 0))


Xtr, ytr = make(30000, w_before, 0.0)
model = LogisticRegression(max_iter=5000).fit(Xtr, ytr)

worlds = {"inputs moved, relationship intact": (w_before, 0.7),
          "the relationship itself moved": (np.array([0.3, -0.9, 1.4]), 0.0)}

print("%-36s %8s %9s %14s %14s"
      % ("after the shift", "AUC", "ECE", "ECE recal", "AUC recal"))
Xh, yh = make(20000, w_before, 0.0)
print("%-36s %8.4f %9.4f %14s %14s"
      % ("no shift", roc_auc_score(yh, model.predict_proba(Xh)[:, 1]),
         ece(yh, model.predict_proba(Xh)[:, 1]), "-", "-"))
for name, (w, mu) in worlds.items():
    Xc, yc = make(20000, w, mu)
    p = model.predict_proba(Xc)[:, 1]
    cal = IsotonicRegression(out_of_bounds="clip").fit(p[:10000], yc[:10000])
    q = cal.predict(p[10000:])
    print("%-36s %8.4f %9.4f %14.4f %14.4f"
          % (name, roc_auc_score(yc[10000:], p[10000:]), ece(yc[10000:], p[10000:]),
             ece(yc[10000:], q), roc_auc_score(yc[10000:], q)))

after the shift                           AUC       ECE      ECE recal      AUC recal
no shift                               0.8147    0.0098              -              -
inputs moved, relationship intact      0.8170    0.0101         0.0103         0.8163
the relationship itself moved          0.7181    0.0830         0.0138         0.7173

The first row is the case worth remembering when someone reports drift: the inputs moved, the relationship did not, and nothing was damaged. Drift in the inputs is not by itself harm.

In the two shifted rows, all four metric columns are computed on the second 10,000 rows, which the calibrator was not fitted on; the no-shift row uses its own 20,000 fresh rows. After the relationship changes, isotonic recalibration reduces calibration error in this simulation but does not recover the original discrimination. An isotonic map is nondecreasing: it can merge scores into ties and change AUC. Only a strictly increasing transformation preserves every ordering and AUC. Evaluate both metrics on held-out data instead of assuming either is unchanged.

When to retrain can be measured

The standard advice after a macro shift is to retrain once enough post-shift data has accumulated, which leaves enough undefined. Where labels from the new regime can be collected, the comparison can be computed. The simulation below is hypothetical: synthetic data in which the new coefficients are generated to correlate at 0.35 in expectation with the old ones (about 0.54 in the 25 coefficients actually drawn), and in which fresh labels are assumed to be plentiful, independent, and representative of current traffic. There are 25 features. The incumbent is a logistic model trained on 40,000 rows of the old world, and every model is scored by AUC on 40,000 held-out rows of the new world. For each number of fresh rows, 15 repetitions compare a model refitted on the fresh rows alone (retrain) with one fitted on the fresh rows plus the same number of old rows (blend), and the last column names the best of incumbent, retrain and blend. The first line of the output gives the incumbent’s AUC and the ceiling reached by a model that knows the new coefficients exactly.

rng = np.random.default_rng(555)
d = 25
w_old = rng.normal(size=d) * 0.5
w_new = 0.35 * w_old + np.sqrt(1 - 0.35 ** 2) * rng.normal(size=d) * 0.5


def make(n, w):
    X = rng.normal(size=(n, d))
    return X, (rng.random(n) < 1 / (1 + np.exp(-X @ w))).astype(int)


Xo, yo = make(40000, w_old)
incumbent = LogisticRegression(max_iter=5000).fit(Xo, yo)
Xt, yt = make(40000, w_new)
base = roc_auc_score(yt, incumbent.predict_proba(Xt)[:, 1])
ceiling = roc_auc_score(yt, Xt @ w_new)

print("incumbent on the new world %.4f, exact new coefficients %.4f" % (base, ceiling))
print("%12s %14s %14s %14s" % ("rows since", "retrain", "blend with old", "best"))
for n in (50, 150, 400, 1000, 3000, 10000):
    fresh, blend = [], []
    for _ in range(15):
        Xn, yn = make(n, w_new)
        fresh.append(roc_auc_score(yt, LogisticRegression(max_iter=5000)
                                   .fit(Xn, yn).predict_proba(Xt)[:, 1]))
        blend.append(roc_auc_score(yt, LogisticRegression(max_iter=5000)
                                   .fit(np.vstack([Xo[:n], Xn]),
                                        np.concatenate([yo[:n], yn]))
                                   .predict_proba(Xt)[:, 1]))
    a, b = np.mean(fresh), np.mean(blend)
    print("%12d %14.4f %14.4f %14s"
          % (n, a, b, max((base, "incumbent"), (a, "retrain"), (b, "blend"))[1]))

incumbent on the new world 0.7092, exact new coefficients 0.9016
  rows since        retrain blend with old           best
          50         0.7825         0.8009          blend
         150         0.8548         0.8426        retrain
         400         0.8841         0.8479        retrain
        1000         0.8943         0.8414        retrain
        3000         0.8996         0.8476        retrain
       10000         0.9010         0.8474        retrain

In this synthetic setting the incumbent does not win a single row. Fifty labeled rows already beat a model fitted to a world that ended, and by a hundred and fifty the fresh model is most of the way to the ceiling. What the comparison measures is not is the new model good but is it better than this one, on held-out labels from the new regime. The 50-row result does not transfer to other systems: it depends on how learnable the problem is, whether the labels are independent and representative, the uncertainty of the comparison itself (the test set here has 40,000 rows, far more than a team may be able to label), the loss the business can tolerate, and the cost of releasing a new model.

In this simulation, keeping the old data as ballast helps in the first row and then holds the model down, plateauing near 0.847 while retraining reaches 0.9010. The cost is not a one-off; here it is paid on every prediction for as long as the stale rows stay in the training set.

In systems with delayed outcomes, the labels needed to draw this curve may arrive weeks after the predictions. That can delay evidence of concept drift; statistical uncertainty, selective labels, and monitoring design also affect what can be established. The baseline a macro shift is measured against is also the thing it invalidates, which is a question about windows rather than about the shift.

References

• Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46(4), 1–37.
• scikit-learn documentation, sklearn.isotonic.IsotonicRegression.