Leading Indicators

Leading indicators are early signals that warn about outcomes before they arrive. They say what might happen rather than confirming what already did, and in machine learning they usually concern the quality and distribution of the inputs rather than the quality of the predictions.

Three properties define them. They are proactive, so there is time to act before performance drops. They are indirect, measuring conditions that affect the result rather than the result itself. And they are sensitive in the short term, moving quickly when something changes.

In practice they come in four families. Data drift — feature distributions moving, new categories appearing. Input quality — a rising missing-value rate, an unexpected schema. Operational signals — latency spikes in the feature pipeline, errors from an upstream source. And representation shift, where the embeddings of user behavior stop resembling the historical ones.

The reason to have them is that they move before the lagging indicators do: transaction patterns in new countries rise this week, and the fraud classifier’s AUC drops next week. The standard account then adds a careful sentence — they signal risk rather than proving failure — and leaves it there.

How loose the link between the signal and the damage is

Drift and harm can come apart

Drift is a change in the inputs; harm is a change in how well the predictions match the outcome. Take a model that scores each case as \(0.9x_1 + 0.4x_2\), with two standard normal features. The true label is 1 when the same combination plus noise (0.6 times a standard normal) is positive, so on reference data the model ranks well. Production data then changes in one of two ways: feature 1 shifts by 0.8 or 1.5 while the rule stays the same, or the inputs stay as they were and the rule itself changes to \(0.2x_1 + 1.1x_2\). For each case, the code reports the KS statistic of feature 1 between reference and production (the drift signal) and the AUC of the model before and after (the harm).

import numpy as np
from scipy.stats import norm, ks_2samp
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
n = 20_000
w = np.array([0.9, 0.4])                      # the model: score = w . x

def run(shift, new_rule):
    x_ref = norm.ppf(rng.random((n, 2)))
    y_ref = (x_ref @ w + 0.6 * norm.ppf(rng.random(n)) > 0)
    x_new = norm.ppf(rng.random((n, 2))) + np.array([shift, 0.0])
    rule = np.array([0.2, 1.1]) if new_rule else w
    y_new = (x_new @ rule + 0.6 * norm.ppf(rng.random(n)) > 0)
    ks = ks_2samp(x_ref[:, 0], x_new[:, 0]).statistic
    return ks, roc_auc_score(y_ref, x_ref @ w), roc_auc_score(y_new, x_new @ w)

for name, shift, new_rule in (("nothing", 0.0, False),
                              ("feature 1 shifts by 0.8, same rule", 0.8, False),
                              ("feature 1 shifts by 1.5, same rule", 1.5, False),
                              ("inputs unchanged, the rule changes", 0.0, True)):
    ks, auc_ref, auc_new = run(shift, new_rule)
    print(f"{name}: KS {ks:.4f}, AUC {auc_ref:.4f} -> {auc_new:.4f}")

nothing: KS 0.0083, AUC 0.9106 -> 0.9126
feature 1 shifts by 0.8, same rule: KS 0.3197, AUC 0.9123 -> 0.9195
feature 1 shifts by 1.5, same rule: KS 0.5524, AUC 0.9124 -> 0.9375
inputs unchanged, the rule changes: KS 0.0083, AUC 0.9115 -> 0.7302

The link breaks in both directions. A large covariate shift — a KS statistic of 0.5524, which any monitor would flag loudly — does not lower the measured AUC in this example, which is 0.9124 before the shift and 0.9375 after it. The rule linking features to labels is unchanged here, so the AUC moved because the mix of cases changed: AUC can differ across populations even when P(Y|X) stays fixed. A change in the relationship itself takes the AUC from 0.9115 to 0.7302 and produces no drift signal at all, because the inputs never moved. Drift is neither necessary nor sufficient for harm, which is the precise version of “they do not prove failure”.

A drift test fires on tiny shifts at large samples

Drift is usually computed with a hypothesis test, so take two samples of \(m\) rows from standard normal distributions, where the second has its mean moved by a true shift of 0.02, 0.10 or 0.50 standard deviations, and record the KS test p-value as \(m\) grows.

print("rows compared | true shift 0.02 | true shift 0.10 | true shift 0.50")
for m in (1_000, 10_000, 100_000, 1_000_000):
    pvalues = []
    for d in (0.02, 0.10, 0.50):
        a = norm.ppf(rng.random(m))
        b = norm.ppf(rng.random(m)) + d
        pvalues.append(ks_2samp(a, b).pvalue)
    print(f"{m:>13,} | " + " | ".join(f"{p:>15.2e}" for p in pvalues))

rows compared | true shift 0.02 | true shift 0.10 | true shift 0.50
        1,000 |        4.33e-01 |        5.31e-03 |        2.05e-24
       10,000 |        1.60e-01 |        3.84e-13 |       3.20e-186
      100,000 |        1.17e-04 |        1.04e-81 |        0.00e+00
    1,000,000 |        1.16e-28 |        0.00e+00 |        0.00e+00

The practical trap is that a KS test is a hypothesis test, so its sensitivity grows with the sample. At a million rows a shift of 0.02 standard deviations returns a p-value of \(10^{-28}\), and a monitor built on a significance threshold fires continuously as soon as traffic grows. A drift monitor has to threshold an effect size, and the size has to be chosen from what the model can tolerate rather than from what the test can detect.

Choosing an alert threshold

The last example is a hypothetical threshold scenario. Before a real degradation, a weekly drift signal rises over eight weeks through 0.02, 0.03, 0.05, 0.08, 0.12, 0.17, 0.23 and 0.30. Each threshold also produces a number of noise alerts a year (26, 9, 4 and 1 for the four thresholds), and two real events happen in the year. The warning signals, noise alerts and real events are specified inputs, not measured annual outcomes. The weeks of warning are the weeks from the first one at or above the threshold to the end of the eight.

signal = np.array([0.02, 0.03, 0.05, 0.08, 0.12, 0.17, 0.23, 0.30])
real_events = 2

for threshold, noise_alerts in ((0.05, 26), (0.10, 9), (0.15, 4), (0.25, 1)):
    first_week = np.argmax(signal >= threshold) + 1
    weeks_of_warning = len(signal) - first_week + 1
    alerts = noise_alerts + real_events
    unit = "week" if weeks_of_warning == 1 else "weeks"
    print(f"threshold {threshold:.2f}: {weeks_of_warning} {unit} of warning, "
          f"{alerts} alerts a year, {real_events / alerts:.1%} of them real")

threshold 0.05: 6 weeks of warning, 28 alerts a year, 7.1% of them real
threshold 0.10: 4 weeks of warning, 11 alerts a year, 18.2% of them real
threshold 0.15: 3 weeks of warning, 6 alerts a year, 33.3% of them real
threshold 0.25: 1 week of warning, 3 alerts a year, 66.7% of them real

Within that scenario, a low threshold gives six weeks of warning and twenty-eight alerts a year, two of which mattered; a high one gives three alerts and one week to act. Neither is correct. The right threshold is a statement about how much investigation time the team has, and it deserves to be set deliberately instead of inherited from a tool’s default.

Using them well

None of this argues against leading indicators. The argument for them is the label delay: where outcomes arrive weeks or months after the prediction, a lagging indicator cannot tell you anything about today, and input checks, output distributions, operational telemetry and timely proxy outcomes may provide earlier signals. That case is strongest exactly where it is usually made — fraud, credit, churn, anything with a slow ground truth.

What the measurements suggest is a different way of choosing them. Prefer signals close to the model’s decision over signals close to the raw data: drift in the predicted score distribution, or in the rate at which the model produces confident predictions, sits nearer the harm than drift in an arbitrary input column. Weight features by importance rather than monitoring all of them equally, while retaining unused fields that help monitor upstream health, selection or vulnerable subgroups; absence from the predictor does not make a field irrelevant to monitoring. And where a leading indicator fires often, measure how often it was followed by real degradation — that number, not the alert count, is what says whether the monitor is worth its investigation cost.

They are the first half of a pair. The second half confirms whether anything actually happened, and the two together are what a model KPI set is made of.

References

• SciPy documentation, scipy.stats.ks_2samp and scipy.stats.norm. docs.scipy.org