Matured-Label Cohort

A matured-label cohort is a group of predictions, usually those made within one time window, that is evaluated only once its labels are mature: enough time has passed for the outcomes to occur and be reported, and the share of labels that have arrived meets a completeness criterion set for the application. The cohort is defined by when the predictions were made, and it is scored once, not on whatever labels exist today.

Why the labels available today mislead

When the outcome is confirmed some time after the prediction, labels arrive in an order that depends on the outcome. A delivery that is on time is confirmed early; a late one takes longer to be delivered and then longer to be reported. Looking at the labels present today therefore overrepresents the quick, usually negative, outcomes. The simulation below ships 60,000 orders over 28 days with a 17 percent late rate. Late orders take longer in transit, and a label reaches the platform shortly after delivery. Python 3.12.14, numpy 2.2.3.

import numpy as np
rng = np.random.default_rng(5)
n = 60000
ship = rng.uniform(0, 28, n)                              # ship day
late = rng.random(n) < 0.17
transit = np.where(late, rng.uniform(4, 8, n), rng.uniform(1, 5, n))   # days; late orders take longer
label_day = ship + transit + rng.uniform(0.2, 1.7, n)     # label reaches the platform 4-40 h after delivery
print(f"true late rate: {late.mean():.3f}   max label delay after shipping: {(label_day-ship).max():.1f} days")
today = 14.0
w = (ship >= 7) & (ship < 14)
avail = w & (label_day <= today)
print(f"day {today:.0f}, orders shipped on days 7-13: {w.sum()}, labels available: {avail.sum()} ({avail.sum()/w.sum():.0%})")
print(f"  late rate among labels available today : {late[avail].mean():.3f}")
mat = w & (label_day <= 14 + 10)
print(f"day 24, cohort 7-13 fully matured: labels {mat.sum()}/{w.sum()}, late rate {late[mat].mean():.3f}  (truth {late[w].mean():.3f})")

true late rate: 0.169   max label delay after shipping: 9.7 days
day 14, orders shipped on days 7-13: 15019, labels available: 5635 (38%)
  late rate among labels available today : 0.037
day 24, cohort 7-13 fully matured: labels 15019/15019, late rate 0.165  (truth 0.165)

On day 14, only 38 percent of the labels for orders shipped on days 7 to 13 have arrived, and among them the late rate is 3.7 percent against a true 16.5 percent. A monitor that scored the rolling window of labels present on day 14 would report an unusually low late rate, and a model scored on those labels would be judged almost entirely on easy negatives. On day 24, ten days after the cohort closed and after the longest label delay in this data, all 15,019 labels are in and the late rate is 16.5 percent, the true value. Because every label in this simulation eventually arrives, maturity here coincides with a complete set of labels.

The rule

• Group by prediction time. A cohort is the predictions made in a window, such as a week, so that late labels update the cohort they belong to.
• Wait for maturity. Score the cohort after the maximum or a high percentile of the label delay has passed since the window closed, or when the fraction of labels received crosses a stated level. The delay distribution is measured from history and checked for change.
• Score once, then keep the result. Re-scoring a cohort as stragglers arrive is fine if the result is labeled as updated, but a decision should rest on the matured result.
• Use the same rule for training. Train only on rows whose labels had arrived by the training cutoff, with features rebuilt as they were at prediction time. When delays are bounded and the split is by time, as in this example, leave a gap between the last training row and the first test row at least as long as the delay, so no training label comes from an outcome unknown when testing begins. Other evaluation designs set the gap from label availability and from any overlap between prediction groups and outcome windows.

The price is lag: a cohort’s accuracy is known about a window plus a delay after the predictions were made. In the example, that is up to 17 days for a weekly cohort. Faster signals that need no labels, such as input distributions and PSI, cover the wait, and a cohort result confirms or contradicts them later.

Limits

A fixed wait does not guarantee completeness. Some outcomes are never reported, and the missing ones may differ from the reported ones, so completion should be measured by cohort and segment, as described under label delay. Waiting also does not create labels for outcomes the system never observes. In the simulation the delay is known exactly and bounded; real delays have long tails, and a cohort labeled as matured at 95 percent completeness can still be missing a systematic subset.

References

• Google, Rules of Machine Learning (Rule 33, test on future data)