Lagging Indicators

A lagging indicator confirms the impact of a change after it has happened. It measures outcomes rather than early signals, and in model monitoring that means the performance metrics themselves — loss, AUC, accuracy, calibration.

Three properties define them, and they are the mirror image of the leading ones. They are reactive, showing a problem only once it has occurred. They measure the end result directly rather than a condition that affects it. And they are what validation and confirmation are built on.

They come in two layers. Model performance: accuracy, precision, recall, F1, AUC, log loss, calibration error. Business outcomes, after the fact: click-through rate falling, fraud losses rising, churn increasing. Performance changes can have several causes; delayed outcome evidence alone does not identify drift as the cause.

They matter because they are the confirmation: they say whether a drift alert or an input-quality problem actually did anything, and they are what a retraining decision is made on. Two things about them are worth measuring rather than asserting — which of them to watch, and how long the lag really is.

Which outcome metric, and how late

Score transformations: AUC stays, calibration moves

Simulate 200,000 cases. Each case has a standard normal score \(s\), and its outcome is 1 with probability \(1/(1+e^{-1.6s})\). Using that probability as the model’s output gives a well-calibrated model, and the same score can be handed to the business in other forms.

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(0)
n = 200_000
score = norm.ppf(rng.random(n))
base = 1 / (1 + np.exp(-1.6 * score))     # the well-calibrated probability
y = rng.random(n) < base                  # the observed outcome

Three lagging measurements are compared. AUC looks only at ranking. Log loss penalizes probabilities that are confidently wrong. Calibration error (expected calibration error, ECE) splits the outputs into 20 equal-width probability bins and averages the gap between the mean predicted probability and the observed outcome rate in each bin, weighted by the share of cases in the bin. The four outputs are the calibrated probability, the probabilities halved, the logit scale doubled, and a shift of 0.15 followed by clipping at 1.

from sklearn.metrics import roc_auc_score

def logloss(pr, lab):
    pr = np.clip(pr, 1e-9, 1 - 1e-9)
    return -np.mean(lab * np.log(pr) + (1 - lab) * np.log(1 - pr))

def ece(pr, lab, bins=20):
    edges = np.linspace(0, 1, bins + 1)
    idx = np.clip(np.digitize(pr, edges[1:-1]), 0, bins - 1)
    total = 0.0
    for b in range(bins):
        in_bin = idx == b
        if in_bin.any():
            total += in_bin.mean() * abs(pr[in_bin].mean() - lab[in_bin].mean())
    return total

outputs = {"well calibrated": base,
           "same ranking, probabilities halved": base * 0.5,
           "same ranking, pushed to extremes": 1 / (1 + np.exp(-3.2 * score)),
           "shifted and clipped, possible ties": np.clip(base + 0.15, 0, 1)}
for name, pr in outputs.items():
    print(f"{name}: AUC {roc_auc_score(y, pr):.4f}, log loss {logloss(pr, y):.4f}, "
          f"calibration error {ece(pr, y):.4f}")

well calibrated: AUC 0.8214, log loss 0.5166, calibration error 0.0038
same ranking, probabilities halved: AUC 0.8214, log loss 0.7025, calibration error 0.2483
same ranking, pushed to extremes: AUC 0.8214, log loss 0.5920, calibration error 0.1028
shifted and clipped, possible ties: AUC 0.8205, log loss 0.7882, calibration error 0.1423

Strictly increasing transformations preserve ranking and AUC while changing calibration. The code evaluates each transformed score with tie-aware AUC. Halving probabilities and changing logit scale keep the AUC at 0.8214 while the calibration error goes from 0.0038 to 0.2483 and 0.1028; adding 0.15 and clipping can create ties and changes the AUC slightly, to 0.8205. Report probability quality separately whenever outputs are used as probabilities.

Label delay and the cost of waiting

The second example uses a separate operating-point quantity, not an AUC. The daily fraud exposure is \$400,000, and the fraction of those dollars recovered falls from 87% to 72%. Until labels arrive and confirm the drop, the extra loss accumulates for every day of the label delay:

\[ \text{extra loss per day} = 400{,}000 \times (0.87 – 0.72) = 60{,}000 \]

Label delay (days)	Recovered per day at 87%	Extra unrecovered loss at 72% over the delay
1	348,000	60,000
7	348,000	420,000
30	348,000	1,800,000
90	348,000	5,400,000

This implies \$60,000 additional unrecovered loss per day under those assumptions. These fractions are not AUCs, and the money cannot be inferred from an AUC change. A 90-day label delay can postpone confirmation; early partial labels require explicit maturity and selection handling.

Comparing two scores on the same cases

The last question is whether one score really beats another when both are evaluated on the same labeled cases. The simulation fixes the class balance at 2% positives and gives each score a binormal distribution: negatives have mean 0 and standard deviation 1, and positives are shifted so that the population AUCs are 0.87 and 0.82, a true difference of 0.05. Within each class the two scores have correlation 0.8, which is what makes the comparison paired.

The paired DeLong method gives each case a placement value. For a positive case it is the share of negatives scored below it (ties count one half), and for a negative case it is the share of positives scored above it. The mean of the positive placement values is the AUC. The covariance of the two scores’ placement values, within positives and within negatives, gives the standard error of the AUC difference.

def paired_delong(pos, neg):
    """AUC(score 0) - AUC(score 1) and its paired standard error."""
    npos, nneg = len(pos), len(neg)
    vp, vn = [], []
    for j in range(2):
        sn, sp = np.sort(neg[:, j]), np.sort(pos[:, j])
        vp.append((np.searchsorted(sn, pos[:, j], 'left') +
                   np.searchsorted(sn, pos[:, j], 'right')) / (2 * nneg))
        vn.append(1 - (np.searchsorted(sp, neg[:, j], 'left') +
                       np.searchsorted(sp, neg[:, j], 'right')) / (2 * npos))
    vp, vn = np.array(vp), np.array(vn)
    covariance = np.cov(vp) / npos + np.cov(vn) / nneg
    contrast = np.array([1., -1.])
    se = np.sqrt(max(0., contrast @ covariance @ contrast))
    return vp[0].mean() - vp[1].mean(), se

cov = [[1, .8], [.8, 1]]
print("cases | positives | AUC difference | paired SE")
for m in (200, 1_000, 5_000, 20_000):
    npos = int(m * 0.02); nneg = m - npos
    # binormal means chosen so the population AUCs are 0.87 and 0.82
    pos = rng.multivariate_normal(np.sqrt(2) * norm.ppf([.87, .82]), cov, npos)
    neg = rng.multivariate_normal([0, 0], cov, nneg)
    difference, se = paired_delong(pos, neg)
    print(f"{m:>5,} | {npos:>9} | {difference:>14.4f} | {se:>9.4f}")

cases | positives | AUC difference | paired SE
  200 |         4 |         0.0000 |    0.0773
1,000 |        20 |         0.1169 |    0.0305
5,000 |       100 |         0.0388 |    0.0132
20,000 |       400 |         0.0468 |    0.0070

These are sample estimates, not universal power values. With 4 positives the estimated difference is 0.0000 with a standard error of 0.0773, far from the true 0.05, and with 20 positives it overshoots to 0.1169. At 400 positives the estimate is 0.0468 with a standard error of 0.0070. Because positives are fixed at 2% of cases here, the two counts grow together in this table; the standard error depends mainly on the number of positives, so adding only negatives to a fixed set of positives barely lowers it. Both class counts and covariance matter, and asymptotic intervals with very few positives can be unreliable.

How to use them

Report a ranking metric and a probability metric together, since the score-transformation example shows they fail independently. Where the model’s output is consumed as a score, ranking and metrics at the deployed operating point are relevant; where it is consumed as a probability, calibration error belongs next to them.

Group results by prediction cohort, and show the evaluation as-of date and label-maturity rule. This places degradation when predictions were made while making delayed evidence visible. Choose actionable effect sizes and assess uncertainty or power using representative paired scores and the actual observation process.

Business outcomes sit one layer further out and lag further still, since click-through rate and churn move for many reasons besides the model. They are the right metrics for deciding whether the model is worth having, and poor ones for detecting that it broke — which is the division of labor a full model KPI set is built around.

References

• SciPy documentation, scipy.stats.norm. docs.scipy.org
• DeLong et al., Comparing areas under correlated ROC curves. Source