Drift Detection
A drift detector compares production with a baseline and raises a flag. Choosing the windows, thresholds, and response is where the design work lies.
-
-
Representation Shift
Neighbor overlap falls to 0.100 while the unbiased embedding MMD estimate remains near zero, in the nearly rotationally symmetric example with old-index/new-query embeddings.
-
Classifier Two-Sample Tests (C2STs)
A random forest scores 1.0000 in training on two samples drawn from the same distribution. Held out, it scores 0.4700.
-
Energy Distance
Record the same ages in days instead of years and the energy distance goes from 0.8126 to 296.7954. The KS reading and the p-value do not move.
-
Maximum Mean Discrepancy (MMD)
Independent samples from the same distribution have a positive expected biased squared-MMD estimate; its null expectation shrinks with sample size.
-
Off-Distribution
A batch that moved as a population with not one unusual row in it, and a batch of unusual rows whose p-value would not trip a 5% alarm.
-
Kolmogorov-Smirnov Test (KS Test)
Moving 1% of the data out to six standard deviations gives D = 0.014000 and p = 0.828022. Not detected.
-
Jensen–Shannon (JS) Divergence
JS reaches 0.693147 nats for disjoint supports. A parameter path that keeps them disjoint has a constant objective, unlike one that changes overlap.