Guardrails in ML

Guardrails are the checks that sit beside a model’s objective rather than inside it: input validation, minimum performance, fairness constraints, latency and cost limits, drift alerts. They are described as a list of categories, each independently sensible, and the list is where the trouble starts. Several properties of a guardrail set do not appear in any of its rows.

A set of rules can be unsatisfiable

Approval-rate parity concerns decisions, while calibration within groups concerns predicted probabilities. They can coexist even with different base rates. Equal positive predictive value among approved customers is a third condition: changing group thresholds can change that selected-pool value without changing the calibration of the underlying scores.

Two equal-sized groups have different input distributions (group 1 is shifted upward on the first feature), and a logistic regression is fitted to the repayment label. The base repayment rate of each group is printed first.

import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(316)
n = 60000
group = (rng.random(n) < 0.5).astype(int)
X = rng.normal(size=(n, 2)) + np.column_stack([0.85 * group, np.zeros(n)])
p_true = 1 / (1 + np.exp(-(0.9 * X[:, 0] + 0.6 * X[:, 1] - 0.4)))
y = (rng.random(n) < p_true).astype(int)
model = LogisticRegression(max_iter=5000).fit(X, y)
score = model.predict_proba(X)[:, 1]

for g in (0, 1):
    print("group %d   base repayment rate %.4f" % (g, y[group == g].mean()))

group 0   base repayment rate 0.4252
group 1   base repayment rate 0.5719

The two policies approve the same number of customers overall but split them differently. one approves every score above 0.55. parity sets a separate cut for each group so that both groups are approved at the overall approval rate of one. For each policy and group the table reports the approval rate, the repayment rate of the approved pool, and reject|bad, the share of defaulters who were rejected.

def metrics(approve, g):
    m = group == g
    return (approve[m].mean(),                 # approval rate
            y[m & approve].mean(),             # repayment rate of the approved pool
            (~approve[m & (y == 0)]).mean())   # share of defaulters rejected


one = score > 0.55
cut = {g: np.quantile(score[group == g], 1 - one.mean()) for g in (0, 1)}
parity = np.where(group == 0, score > cut[0], score > cut[1])

print("%-24s %5s %9s %9s %11s" % ("policy", "group", "approve", "repay", "reject|bad"))
for label, approve in (("one threshold for all", one), ("equal approval rates", parity)):
    for g in (0, 1):
        print("%-24s %5d %9.4f %9.4f %11.4f" % ((label, g) + metrics(approve, g)))

gap_one = abs(one[group == 0].mean() - one[group == 1].mean()) * 100
gap_par = abs(y[(group == 0) & parity].mean() - y[(group == 1) & parity].mean()) * 100
print("approval-rate gap under one threshold: %.1f points" % gap_one)
print("repayment gap between approved pools under equal approval rates: %.1f points" % gap_par)

policy                   group   approve     repay  reject|bad
one threshold for all        0    0.2978    0.6928      0.8409
one threshold for all        1    0.5574    0.7327      0.6520
equal approval rates         0    0.4284    0.6373      0.7296
equal approval rates         1    0.4284    0.7752      0.7751
approval-rate gap under one threshold: 26.0 points
repayment gap between approved pools under equal approval rates: 13.8 points

The first policy uses one score and one bar. Its two approved pools repay within four points of each other (0.6928 and 0.7327), and the approval rates differ by 26.0 points because the two groups do not carry the same base repayment rate (0.4252 and 0.5719), so a guardrail requiring approval rates within three points of each other fails it on day one. The second policy closes that gap by construction and opens a 13.8-point gap in what an approval means: under equal approval rates the approved pools repay 0.6373 and 0.7752.

Kleinberg, Mullainathan and Raghavan consider three conditions together: calibration within groups and balance of average scores for each of the positive and negative outcome classes. Except for equal base rates or perfect prediction, those three cannot all hold. This is not an impossibility theorem for demographic parity plus score calibration. Related predictive-parity versus error-rate results have their own nondegeneracy conditions; match each theorem to the exact constraints being imposed.

A guardrail sheet should state each fairness criterion precisely and check whether the particular set is jointly feasible. When requirements conflict under the relevant assumptions, choose and document the trade-off rather than invoking an unrelated impossibility result.

Guardrails compound

Each check is specified on its own, at its own level. The pager is connected to all of them at once.

The examples that follow continue in the same Python session as the first, so NumPy is not imported again. simulate draws 365 days of k standard-normal test statistics that share a common component of correlation rho, raises an alarm on a day when any statistic exceeds the two-sided critical value for level alpha, and averages the share of alarm days and the mean gap between alarm days over 400 repetitions. The last row is the Bonferroni level, alpha divided by six.

from scipy.stats import norm

rng = np.random.default_rng(5150)
days, reps = 365, 400


def simulate(k, alpha, rho):
    gate = norm.ppf(1 - alpha / 2)
    hit_days, gaps = [], []
    for _ in range(reps):
        common = rng.normal(size=(days, 1))
        z = (np.sqrt(rho) * common
             + np.sqrt(1 - rho) * rng.normal(size=(days, k)))
        fired = (np.abs(z) > gate).any(axis=1)
        hit_days.append(fired.mean())
        idx = np.flatnonzero(fired)
        gaps.append(np.mean(np.diff(idx)) if len(idx) > 1 else float(days))
    return np.mean(hit_days), np.mean(gaps)


print("%6s %10s %8s %18s %20s"
      % ("checks", "level", "corr", "days with an alarm", "days between alarms"))
for k, alpha, rho in ((1, 0.05, 0.0), (3, 0.05, 0.0), (6, 0.05, 0.0),
                      (6, 0.05, 0.5), (6, 0.05 / 6, 0.0)):
    rate, gap = simulate(k, alpha, rho)
    print("%6d %10.4f %8.1f %17.1f%% %19.1f" % (k, alpha, rho, rate * 100, gap))
print("1 - 0.95**6 = %.1f%%" % ((1 - 0.95 ** 6) * 100))

checks      level     corr days with an alarm  days between alarms
     1     0.0500      0.0               5.0%                20.2
     3     0.0500      0.0              14.2%                 7.0
     6     0.0500      0.0              26.4%                 3.8
     6     0.0500      0.5              20.9%                 4.8
     6     0.0083      0.0               4.8%                21.2
1 - 0.95**6 = 26.5%

Adding independent 5% checks raises the share of days with an alarm from 5.0% (one check) to 14.2% (three) and 26.4% (six), and shortens the mean gap between alarms from 20.2 days to 3.8. Six independent 5% checks have familywise false-alarm probability 1−0.95^6, about 26.5% per evaluation (the line printed after the table); the simulation gives 26.4%. Dependence matters: six identical checks still have 5% familywise error, and with a common correlation of 0.5 the six checks alarm on 20.9% of days. Bonferroni uses alpha/6 for each check, brings the alarm rate back to 4.8% in this simulation, and controls the familywise level without independence, but it does not divide statistical power by six. Power loss depends on the test, effect and sample size.

Specify whether the error budget covers one evaluation, a family of checks, or repeated monitoring over time. Adding checks can raise familywise error unless the policy accounts for dependence and multiplicity. Compare false alarms with power for the failures the system must detect.

A guardrail that edits data edits the evidence

Data guardrails do not just observe. Clamping, defaulting and imputation rewrite the rows, and everything downstream reads the rewritten version.

The clamp value is the 99th percentile of the training sample. Each production month is compared with a baseline month using the KS statistic, once on the raw values and once after both samples are clamped at that value.

def ks_stat(a, b):
    a, b = np.sort(a), np.sort(b)
    g = np.concatenate([a, b])
    return float(np.max(np.abs(np.searchsorted(a, g, "right") / a.size
                               - np.searchsorted(b, g, "right") / b.size)))


rng = np.random.default_rng(909)
n = 20000
train = np.exp(rng.normal(3.0, 0.8, 60000))
cap = np.quantile(train, 0.99)
base = np.exp(rng.normal(3.0, 0.8, n))

print("clamp value, the 99th percentile of the training data: %.1f" % cap)
print("%-20s %10s %11s %11s %10s %11s"
      % ("production month", "clamp rate", "raw mean", "clamped", "raw KS", "clamped KS"))
print("%-20s %10s %11.2f %11.2f %10s %11s"
      % ("baseline", "-", base.mean(), np.minimum(base, cap).mean(), "-", "-"))
for label, mu, sd in (("unchanged", 3.0, 0.8), ("mean up 0.3", 3.3, 0.8),
                      ("tail fattened", 3.0, 1.2)):
    cur = np.exp(rng.normal(mu, sd, n))
    print("%-20s %9.2f%% %11.2f %11.2f %10.4f %11.4f"
          % (label, (cur > cap).mean() * 100, cur.mean(), np.minimum(cur, cap).mean(),
             ks_stat(base, cur), ks_stat(np.minimum(base, cap), np.minimum(cur, cap))))

clamp value, the 99th percentile of the training data: 128.2
production month     clamp rate    raw mean     clamped     raw KS  clamped KS
baseline                      -       27.73       27.24          -           -
unchanged                 0.99%       27.56       27.14     0.0044      0.0044
mean up 0.3               2.67%       37.36       35.88     0.1484      0.1484
tail fattened             6.21%       41.42       33.98     0.0981      0.0981

The raw and clamped KS readings happen to agree in this example. KS is invariant to the same strictly monotone transformation of both samples; clipping is not strictly monotone and can erase differences confined to clipped tails. Keep raw-value, clamped-value and clamp-rate monitoring distinct. The KS statistic on censored values does not recover removed information.

In the tail-fattened month the clamp hides about half of the increase in the mean (27.73 to 41.42 raw, 27.24 to 33.98 clamped), while the clamp rate rises from about 1% to 6.21% and records how much was rewritten. Any guardrail that rewrites inputs should publish how often it fired, or the monitors after it are reading a censored series without being told. The same applies to a re-scoring stage, which edits outputs for reasons the monitor cannot see.

A threshold also needs a sampling model

For a rate measured from n independent Bernoulli events with probability p, Var(p-hat) = p(1−p)/n. At p = 0.05 the standard deviation is about 4.36 percentage points for 25 observations and 0.49 points for 2,000. A fixed observed-rate threshold therefore has different false-alarm behavior at different volumes. This is an elementary sampling calculation, separate from the multiple-testing adjustment above. For small counts use an appropriate binomial method; for clustered outages or changing event probabilities, model that dependence or variation rather than assuming independent draws.

References

• Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017). Inherent trade-offs in the fair determination of risk scores. Innovations in Theoretical Computer Science (ITCS).
• Chouldechova, A. (2017). Fair prediction with disparate impact: a study of bias in recidivism prediction instruments. Big Data, 5(2), 153–163.