Monitoring, Drift, and Retraining

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

d, n_per, T = 4, 2000, 40
def week(t, gg):
    X = gg.normal(size=(n_per, d))
    w = np.array([1.2, -0.8, 0.5, 0.0]) + t * np.array([-0.06, 0.04, 0.0, 0.0])
    y = (gg.random(n_per) < 1 / (1 + np.exp(-(X @ w)))).astype(int)
    return X, y
weeks = [week(t, np.random.default_rng(100 + t)) for t in range(T)]

for cadence in (0, 1, 4, 12):                       # 0 = never retrain
    model = LogisticRegression(max_iter=2000).fit(*weeks[0])
    hist_X, hist_y = weeks[0]
    aucs = []; refits = 0
    for t in range(1, T):
        Xt, yt = weeks[t]
        aucs.append(roc_auc_score(yt, model.predict_proba(Xt)[:, 1]))
        hist_X = np.vstack([hist_X, Xt])[-8000:]     # a rolling window
        hist_y = np.concatenate([hist_y, yt])[-8000:]
        if cadence and t % cadence == 0 and t < T - 1:
            model = LogisticRegression(max_iter=2000).fit(hist_X, hist_y)
            refits += 1
    label = "never" if cadence == 0 else f"every {cadence} week(s)"
    print(f"retrain {label:16s} mean AUC {np.mean(aucs):.4f}  final {aucs[-1]:.4f}"
          f"  worst {min(aucs):.4f}  refits {refits}")
# retrain never            mean AUC 0.5366  final 0.2768  worst 0.2683  refits 0
# retrain every 1 week(s)  mean AUC 0.7112  final 0.8023  worst 0.6112  refits 38
# retrain every 4 week(s)  mean AUC 0.7064  final 0.8004  worst 0.5946  refits 9
# retrain every 12 week(s) mean AUC 0.6835  final 0.8004  worst 0.5007  refits 3

The initial fit uses week 0. Each subsequent week is scored before its data are added to the training window; a refit at the end of a week is available for the following week. This assumes labels arrive by that week’s end and fitting is completed before the next prediction batch. No refit is counted after the final evaluation, because it would serve no evaluated predictions.

Weekly updates use 38 refits and a four-week cadence uses 9, with mean AUCs 0.7112 and 0.7064 in this path. That is about 76% fewer refit calls, not a measured 76% reduction in total cost. Window sizes, labeling, review, and serving costs also matter. “Every four weeks” and “every twelve weeks” are the simulated schedules, not calendar months and quarters.

The twelve-week schedule has worst-week AUC 0.5007, compared with 0.5946 for four-week updates. The mean hides this difference, but neither a sample minimum nor one trajectory establishes a universal cadence. The full-data label mechanism also becomes less separable near the middle of this simulation; not every low AUC is avoidable model staleness. Inspect paired trajectories and relevant decision costs, and repeat the simulation or use historical backtests before choosing a policy.

A schedule can be combined with an outcome-based trigger, but the trigger itself needs a noise tolerance, minimum counts, label-delay policy, and response plan. A triggered refit is a candidate for evaluation, not an automatic production replacement.

import numpy as np
from scipy.stats import ks_2samp
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, brier_score_loss

d = 4
gg = np.random.default_rng(0)
X0 = gg.normal(size=(20000, d))
w0 = np.array([1.2, -0.8, 0.5, 0.0])
y0 = (gg.random(20000) < 1 / (1 + np.exp(-(X0 @ w0)))).astype(int)
m = LogisticRegression(max_iter=2000).fit(X0, y0)
p0 = m.predict_proba(X0)[:, 1]

print(f"{'drift':>6} {'KS inputs':>11} {'KS scores':>11} {'mean score':>11} {'AUC':>8} {'Brier':>8}")
for alpha in (0.0, 0.25, 0.5, 1.0, 2.0):
    X1 = gg.normal(size=(20000, d))
    w1 = w0 + alpha * np.array([-1.2, 0.8, 0.0, 0.0])
    y1 = (gg.random(20000) < 1 / (1 + np.exp(-(X1 @ w1)))).astype(int)
    p1 = m.predict_proba(X1)[:, 1]
    ks_in = min(ks_2samp(X0[:, j], X1[:, j]).pvalue for j in range(d))
    print(f"{alpha:6.2f} {ks_in:11.2e} {ks_2samp(p0, p1).pvalue:11.2e} {p1.mean():11.4f}"
          f" {roc_auc_score(y1, p1):8.4f} {brier_score_loss(y1, p1):8.4f}")
#  drift   KS inputs   KS scores  mean score      AUC    Brier
#   0.00    4.11e-01    4.94e-01      0.5015   0.8170   0.1747
#   0.25    1.20e-01    2.79e-01      0.4998   0.7719   0.1962
#   0.50    3.44e-01    3.25e-01      0.5056   0.7105   0.2258
#   1.00    2.38e-01    6.75e-01      0.5039   0.5385   0.3082
#   2.00    5.32e-02    8.41e-01      0.5043   0.2614   0.4404

The input and score p-values fluctuate across independent samples; their magnitudes do not measure the severity of conditional drift. In this run none of the unadjusted input minima crosses 0.05, while AUC drops from 0.8170 to 0.2614 and Brier loss rises from 0.1747 to 0.4404. A fixed model’s mean score can remain near 0.5 during that change.

For this pure conditional shift, the full input process is unchanged and the prediction rule is fixed. Consequently, the joint distribution of the observed inputs and predictions is unchanged too. A statistic using only those observations cannot distinguish the two label mechanisms beyond its false-alarm behavior. Concept drift more generally can accompany input drift, proxy changes, or changes in temporal dependence, which may supply detectable evidence under additional assumptions.

The minimum of four marginal p-values is shown descriptively here; it is not one calibrated 0.05-level test across all features and times. The prediction reference also uses the training inputs, so these score p-values are not a prospective test calibrated using an independently collected reference. The inability to observe this isolated label change follows from the generating setup, not from accepting these particular null tests.

The precision of a live AUC estimate depends on the number of labeled positives and negatives, score distributions, sampling design, and dependence. A few hundred labels do not guarantee a standard error of 0.02 or a one-week detection time. Plan the label budget with pilot resampling or a relevant variance calculation, include outcome delay, and verify the complete alert procedure over repeated windows.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

d, n_per, T = 4, 4000, 12
def make(seed, nonlinear):
    gg = np.random.default_rng(seed)
    X = gg.normal(size=(n_per, d))
    lin = X @ np.array([1.2, -.8, .5, .3])
    if nonlinear:
        lin -= 1.8*X[:, 0]**2
    y = (gg.random(n_per) < 1/(1+np.exp(-lin))).astype(int)
    return X, y

for nonlinear in (False, True):
    for fraction in (.50, .15):
        X0, y0 = make(0, nonlinear)
        models = {name: LogisticRegression(max_iter=2000).fit(X0, y0)
                  for name in ("selected", "random", "full")}
        history = []; skipped = {name: 0 for name in models}
        for t in range(1, T):
            Xt, yt = make(t, nonlinear)
            scores = {name: model.predict_proba(Xt)[:, 1] for name, model in models.items()}
            history.append([roc_auc_score(yt, scores[name]) for name in models])
            if t == T-1:
                continue
            count = int(round(fraction*n_per))
            chosen = np.lexsort((np.arange(n_per), -scores["selected"]))[:count]
            random_rows = np.random.default_rng(1000+t).choice(n_per, count, replace=False)
            for name, rows in (("selected", chosen), ("random", random_rows), ("full", np.arange(n_per))):
                if np.unique(yt[rows]).size < 2:
                    skipped[name] += 1
                    continue
                models[name] = LogisticRegression(max_iter=2000).fit(Xt[rows], yt[rows])
        history = np.array(history)
        print(f"nonlinear={nonlinear} label fraction={fraction:.2f}")
        print("mean AUC selected/random/full", np.round(history.mean(axis=0), 4))
        print("final AUC selected/random/full", np.round(history[-1], 4))
        print("skipped one-class fits", list(skipped.values()))
# nonlinear=False label fraction=0.50
# mean AUC selected/random/full [0.8167 0.8164 0.8169]
# final AUC selected/random/full [0.8105 0.8097 0.8108]
# skipped one-class fits [0, 0, 0]
# nonlinear=False label fraction=0.15
# mean AUC selected/random/full [0.8123 0.8147 0.8169]
# final AUC selected/random/full [0.8057 0.8091 0.8108]
# skipped one-class fits [0, 0, 0]
# nonlinear=True label fraction=0.50
# mean AUC selected/random/full [0.714  0.7205 0.7211]
# final AUC selected/random/full [0.7215 0.7236 0.7252]
# skipped one-class fits [0, 0, 0]
# nonlinear=True label fraction=0.15
# mean AUC selected/random/full [0.5871 0.7179 0.7211]
# final AUC selected/random/full [0.656  0.7239 0.7252]
# skipped one-class fits [0, 0, 0]

The score-selected and random-label models receive the same number of rows from the same latest batch. The full-label model is an oracle comparator with a larger labeling budget, also trained on that batch. Every batch is evaluated before updating; the synthetic experiment can reveal labels for all rows to compute those evaluation AUCs, although a real selective-label system generally cannot.

When selection depends only on X, it preserves the conditional outcome law within the selected region. A correctly specified, identifiable conditional model can therefore remain consistent under suitable sampling and fitting conditions. Finite samples, regularization, lost coverage, and limited information can still affect it. A flexible model class does not by itself establish these conditions.

With a misspecified linear logit, changing the training distribution changes which approximation is fitted. For the nonlinear generator and 15% labeling budget, the mean AUC is 0.5871 with score selection and 0.7179 with random selection; final AUCs are 0.6560 and 0.7239. With the linear generator at the same budget, mean AUCs are 0.8123 and 0.8147. Comparing the matched-budget random model helps assess selection separately from sample size and recency in this trajectory. The approved fraction is fixed each round; the selected region may move, but it is not necessarily nested or shrinking. Report mean and final AUCs and repeat across trajectories before attributing a general effect to the policy.

Coverage of excluded regions requires a deliberate outcome-acquisition design. Random auditing may help when outcomes can be obtained without changing the action; when outcomes depend on approval, appropriate experiments or additional identification assumptions may be needed. Overriding decisions is not a cost-free labeling operation, and its consequences cannot be inferred from the sampled fraction alone.