Monitoring, Drift, and Retraining

Inputs and predictions can often be inspected before outcomes arrive. Their distributions provide useful operational signals, but a change in those signals is not the same as a change in predictive performance. The examples below separate these questions and show why a retraining policy needs outcome measurement as well as drift detection.

Write \(P_t(X,Y)\) for the data distribution at time \(t\). Covariate shift changes \(P(X)\) while keeping \(P(Y\mid X)\) fixed. Here concept drift means a change in \(P(Y\mid X)\); it may occur with or without input drift. Label shift usually refers to changing \(P(Y)\) with \(P(X\mid Y)\) fixed, a different assumption. These are population changes; a monitor sees finite windows whose statistics also fluctuate.

Before statistical drift tests, check whether the service is producing valid data: schema and units, missing features, stale timestamps, join failures, request volume, latency, and error rates. A preprocessing change can alter model inputs without any change in the underlying population. Compare model and feature versions when investigating an alert.

The alarm and the harm are different things

import numpy as np
from scipy.stats import ks_2samp
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

g = np.random.default_rng(0); n = 20000
Xtr = g.normal(size=(n, 4))
w = np.array([1.2, -0.8, 0.5, 0.0])
ytr = (g.random(n) < 1 / (1 + np.exp(-(Xtr @ w)))).astype(int)
m = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
auc = lambda X, y: roc_auc_score(y, m.predict_proba(X)[:, 1])
def drift(X):
    tests = [ks_2samp(Xtr[:, j], X[:, j]) for j in range(4)]
    return max(r.statistic for r in tests), min(1., 4*min(r.pvalue for r in tests))
ref_rng = np.random.default_rng(991)
Xref = ref_rng.normal(size=(n, 4))
yref = (ref_rng.random(n) < 1/(1+np.exp(-(Xref @ w)))).astype(int)
print(f"reference AUC {auc(Xref, yref):.4f}")
print(f"{'scenario':34s} {'max KS D':>10} {'Bonferroni p':>15} {'AUC':>8}")

Xa = g.normal(loc=[1.0, -1.0, 0.5, 0.0], size=(n, 4))          # inputs moved
ya = (g.random(n) < 1 / (1 + np.exp(-(Xa @ w)))).astype(int)   # relationship unchanged
print(f"{'A covariate shift, same P(y|x)':34s} {drift(Xa)[0]:10.4f} {drift(Xa)[1]:15.3e} {auc(Xa, ya):8.4f}")

Xb = g.normal(size=(n, 4))                                     # same input population
w2 = np.array([-1.2, 0.8, 0.5, 0.0])                           # relationship flipped
yb = (g.random(n) < 1 / (1 + np.exp(-(Xb @ w2)))).astype(int)
print(f"{'B concept drift, same P(x)':34s} {drift(Xb)[0]:10.4f} {drift(Xb)[1]:15.3e} {auc(Xb, yb):8.4f}")

Xc = g.normal(loc=0.03, size=(n, 4))                           # small shift in input means
yc = (g.random(n) < 1 / (1 + np.exp(-(Xc @ w)))).astype(int)
print(f"{'C tiny shift (mean +0.03)':34s} {drift(Xc)[0]:10.4f} {drift(Xc)[1]:15.3e} {auc(Xc, yc):8.4f}")
# reference AUC 0.8203
# scenario                             max KS D    Bonferroni p      AUC
# A covariate shift, same P(y|x)         0.3801       0.000e+00   0.8293
# B concept drift, same P(x)             0.0119       4.785e-01   0.2527
# C tiny shift (mean +0.03)              0.0204       1.915e-03   0.8175

The KS statistic \(D\) is the largest vertical gap between two empirical cumulative distribution functions. A value 0.10 means a ten-percentage-point difference at some cutoff; it does not mean a ten-percent performance loss. The block reports the largest marginal \(D\) and a Bonferroni-adjusted minimum p-value across four features. Row A contains real input drift, so rejecting equality is not a statistical false alarm. Its AUC is 0.8293 versus the independent reference’s 0.8203 in this run, which shows that this change need not reduce ranking performance. It does not prove that all decision costs or subgroup outcomes improved. A printed p-value of zero is a numerical result, not a probability claim of absolute certainty.

Row B keeps the input-generating distribution fixed while changing the label mechanism. The AUC near 0.25 indicates that positive cases tend to rank below negative cases; it does not establish how confident or calibrated the probabilities are. A non-significant KS result is compatible with the unchanged input population, but is not proof of equality, and false positives remain possible.

Row C shifts each input mean by 0.03. Its KS test can be significant with 20,000 rows even though the measured AUC stays near the reference. Differences between these sample AUCs include sampling noise; they are not an exact causal cost of the mean shift. An alert rule should state its false-alarm budget, effect-size requirement, persistence requirement, and the action it triggers.

Sample size and the meaning of an alert

print(f"{'n':>8} " + " ".join(f"{d:>10}" for d in (0.00, 0.01, 0.03, 0.10)))
for nn in (1000, 10000, 100000, 500000):
    row = []
    for d in (0.00, 0.01, 0.03, 0.10):
        gg = np.random.default_rng(1)
        a = gg.normal(size=nn); b = gg.normal(loc=d, size=nn)
        row.append(ks_2samp(a, b).pvalue)
    print(f"{nn:8d} " + " ".join(f"{v:10.2e}" for v in row))
#        n        0.0       0.01       0.03        0.1
#     1000   1.34e-01   1.21e-01   8.69e-02   1.45e-03
#    10000   8.90e-01   8.63e-01   1.19e-01   2.21e-08
#   100000   4.68e-01   5.99e-03   9.04e-11   7.36e-82
#   500000   2.94e-01   3.00e-08   4.03e-42   0.00e+00

For a shift of 0.01 standard deviations, this run produces p=0.86 at n=10,000 and about 3e-8 at n=500,000. The two-sample KS null is equality of the distributions, not a general test of zero mean shift. Larger samples give power against smaller differences. Under an unchanged continuous distribution and independent sampling, a correctly calibrated fixed-level test does not reject continuously merely because n grows. The d=0 column illustrates null sampling variation. This table is one realization per setting, not a power study.

Use magnitude and statistical uncertainty together. The KS statistic, a mean change in reference-standard-deviation units, or a Wasserstein distance can quantify different aspects of change; none is a universal measure of harm. PSI also depends on bin choices and treatment of empty bins. A per-window Bonferroni correction covers the four feature comparisons here, not repeated tests over time. Backtest the full alert rule on historical stable periods and known changes, accounting for overlapping windows, seasonality, and dependent rows. Marginal tests can miss a joint change: two normal features can keep their individual distributions while their correlation changes.

Input and score monitors then have no information about that isolated change. Outcome metrics can reveal its performance consequences. Conversely, a distribution change need not change a particular metric, so an unchanged AUC does not certify that the entire system is unchanged.

Maintain a path to representative outcome measurement. Log prediction-time IDs, model version, scores, decisions, and timestamps so labels can be joined to the correct predictions after a defined outcome window. Compare cohorts at similar label maturity and inspect missing-label rates by group. A random sample of traffic can support evaluation if outcomes are actually ascertained without selective loss; it does not automatically make a finite-sample AUC unbiased or guarantee cheap, timely precision. If outcomes exist or are observable only after an action, sampling alone cannot recover the unobserved counterfactual.

From an alert to a retraining decision

A monitor needs a reference window, a current window, minimum sample and class counts, and an owner for the response. A fixed reference helps expose accumulated change; a rolling reference follows recent conditions but can gradually absorb degradation. Match seasonal periods where relevant. Track task loss or calibration as well as ranking, decision outcomes and important subgroups; undefined metrics from a one-class window require an explicit status, not an invented zero.

An alert starts an investigation. Verify data contracts, model rollout, label maturity, and the affected segments before fitting a new model. Repairing a feature pipeline or rolling back a deployment may be the appropriate response. For retraining, choose a window and weighting scheme using only labels available at that time. Recent data can adapt faster but contain fewer examples; older data can preserve coverage while diluting a new relationship.

Evaluate the candidate against the current model on later, untouched and sufficiently labeled data, using the performance, subgroup, latency, and resource criteria specified for promotion. Backtest a proposed cadence or trigger on earlier periods instead of choosing it from the final test timeline. Deploy through an appropriate staged or shadow evaluation, retain versioned artifacts and a rollback path, and monitor the candidate after promotion. The simulations below compare learning rules; they do not implement this deployment approval process.

References

• SciPy: Two-sample Kolmogorov–Smirnov test
• Wang and Abraham, Concept Drift Detection for Streaming Data