Fairness and Bias Auditing

A fairness audit asks how a system’s decisions distribute benefits, errors, and harms across people. Group metrics make some disparities measurable, but choosing a metric requires knowing what the decision and label mean. Some criteria conflict under specific conditions; others can hold together. This article separates those conditions and shows how measurement choices affect an audit.

What each criterion measures

Let \(A\) denote a group, \(Y\in\{0,1\}\) the evaluation label, \(S\) a score, and \(\hat Y\) the binary decision after applying a threshold. The base rate is \(\pi_a=P(Y=1\mid A=a)\). Selection rate counts positive decisions among everyone; TPR counts them among actual positives; FPR counts them among actual negatives; PPV counts actual positives among positive decisions. For example, 30 true positives and 10 false positives in a group of 100 with 50 actual positives give selection 0.40, TPR 0.60, FPR 0.20, and PPV 0.75. Whether a positive decision allocates a benefit or imposes a burden changes which errors matter.

• Demographic parity: the selection rate is equal across groups.
• Equalized odds: true and false positive rates are equal across groups.
• Predictive parity: positive predictions have equal positive predictive value (PPV) across groups. This is a property of thresholded decisions.

Equal opportunity requires equal TPR only; it is weaker than equalized odds. Calibration within groups instead concerns scores: \(P(Y=1\mid S=s,A=a)=s\). It does not require equal PPV after thresholding, since the selected score distributions may differ. A disparity is a diagnostic finding, not by itself a causal explanation or a complete judgment of fairness.

The arithmetic can be checked directly. If both groups have common TPR \(t\) and FPR \(f\), then selection is \(t\pi_a+f(1-\pi_a)\) and PPV is \(t\pi_a/[t\pi_a+f(1-\pi_a)]\), when defined. With unequal base rates, equalized odds and demographic parity require \(t=f\), an uninformative decision in this sense. Equalized odds with \(t>0,f>0\) gives different PPVs. For example, \(t=0.8,f=0.2\) and base rates 0.3 and 0.6 give selection 0.38 and 0.56, and PPV about 0.632 and 0.857.

These conditions matter. A perfect predictor has equalized odds and PPV 1 in both groups even when base rates differ. An independent fair coin has equalized odds and demographic parity. Neither example supports a rule that only one criterion can hold. The simulation below starts with a score calibrated in the generating population and changes only decision thresholds. Quantile thresholds are chosen and measured on the same rows to illustrate the mechanics; their near-zero sample gaps are not held-out guarantees. Ties or discrete scores can prevent exact quantile targets.

import numpy as np

g = np.random.default_rng(0); n = 400_000
A = (g.random(n) < 0.5).astype(int)                      # the protected attribute
base = np.where(A == 0, 0.30, 0.60)                      # group-specific locations before clipping
p_true = np.clip(base + 0.25 * g.normal(size=n), 0.01, 0.99)
Y = (g.random(n) < p_true).astype(int)
S = p_true                                                # calibrated in the generating population

def audit(pred, name):
    rows = []
    for a in (0, 1):
        m = A == a
        rows.append((pred[m].mean(),
                     pred[m][Y[m] == 1].mean(),
                     pred[m][Y[m] == 0].mean(),
                     Y[m][pred[m] == 1].mean()))
    print(f"{name:22s} " + "  ".join(
        f"g{a}: sel {r[0]:.3f} TPR {r[1]:.3f} FPR {r[2]:.3f} PPV {r[3]:.3f}"
        for a, r in enumerate(rows)))
    return rows

print(f"base rates: group0 {Y[A == 0].mean():.4f}   group1 {Y[A == 1].mean():.4f}")

r = audit((S >= 0.5).astype(int), "one threshold 0.50")
print(f"   selection gap {abs(r[0][0]-r[1][0]):.4f}  TPR gap {abs(r[0][1]-r[1][1]):.4f}"
      f"  FPR gap {abs(r[0][2]-r[1][2]):.4f}  PPV gap {abs(r[0][3]-r[1][3]):.4f}")

target = (S >= 0.5).mean()                                # target selection fraction
th = {a: np.quantile(S[A == a], 1 - target) for a in (0, 1)}
r = audit(np.where(A == 0, S >= th[0], S >= th[1]).astype(int), "equal selection rate")
print(f"   selection gap {abs(r[0][0]-r[1][0]):.4f}  TPR gap {abs(r[0][1]-r[1][1]):.4f}"
      f"  PPV gap {abs(r[0][3]-r[1][3]):.4f}")

tpr_target = ((S >= 0.5)[Y == 1]).mean()                  # target true-positive fraction
th2 = {a: np.quantile(S[(A == a) & (Y == 1)], 1 - tpr_target) for a in (0, 1)}
r = audit(np.where(A == 0, S >= th2[0], S >= th2[1]).astype(int), "equal TPR")
print(f"   TPR gap {abs(r[0][1]-r[1][1]):.4f}  FPR gap {abs(r[0][2]-r[1][2]):.4f}"
      f"  PPV gap {abs(r[0][3]-r[1][3]):.4f}")
# base rates: group0 0.3158   group1 0.5934
# one threshold 0.50     g0: sel 0.213 TPR 0.431 FPR 0.112 PPV 0.639  g1: sel 0.656 TPR 0.806 FPR 0.436 PPV 0.730
#    selection gap 0.4424  TPR gap 0.3750  FPR gap 0.3231  PPV gap 0.0908
# equal selection rate   g0: sel 0.435 TPR 0.724 FPR 0.301 PPV 0.526  g1: sel 0.435 TPR 0.594 FPR 0.202 PPV 0.811
#    selection gap 0.0000  TPR gap 0.1294  PPV gap 0.2856
# equal TPR              g0: sel 0.393 TPR 0.676 FPR 0.262 PPV 0.543  g1: sel 0.514 TPR 0.676 FPR 0.277 PPV 0.781
#    TPR gap 0.0000  FPR gap 0.0146  PPV gap 0.2376

Equalizing selection reduces its displayed gap to 0.0000 while increasing the PPV gap from 0.0908 to 0.2856. Equalizing TPR reduces the TPR gap to 0.0000 and also reduces the FPR gap from 0.3231 to 0.0146, although the PPV gap grows to 0.2376. Thus a change can improve more than one metric; it does not necessarily worsen every criterion except its target. These are rounded finite-sample results.

The score is the Bernoulli probability used to draw \(Y\), so it is calibrated in the population by construction. Empirical outcome frequencies in score bins still fluctuate, and bins can contain different average scores. Changing the threshold leaves this score calibration unchanged. The code measures binary-decision rates, not a theorem about calibration and distributions of continuous scores. Clipping the generating probabilities also means the realized base rates are not exactly the location parameters 0.30 and 0.60.

Choosing a target requires input from affected people and those accountable for deployment, alongside technical evidence. Report the relevant rates, their denominators, uncertainty, and the consequences of errors. Reaching one parity target does not establish that the label, deployment, or treatment of individuals is appropriate.

Auditing a benchmark with and without a group feature

Adult is a historical benchmark for predicting recorded annual income above $50,000. Its sampling, eligibility filters, recorded categories, and income label limit what the audit represents. Base rates reflect that data-production process; they do not determine every possible model disparity. The recorded sex categories are Male and Female and do not represent all gender identities. This is a benchmark classification exercise, not an evaluation of who deserves a benefit.

The next block needs Fairlearn, pandas, and scikit-learn; fetch_adult downloads data on first use. It reserves 30% of rows before fitting the category encoder, and tells the boosted trees which encoded columns are categorical. Unknown categories are treated as missing by the model. The two models share a split and a fixed 0.5 threshold. Group labels remain available for evaluation even when excluded from model inputs. Rows are equally weighted here; fnlwgt is retained as a feature, not used as a survey weight, so the results are not population-weighted estimates.

import numpy as np
from fairlearn.datasets import fetch_adult
from sklearn.model_selection import train_test_split
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.preprocessing import OrdinalEncoder

d = fetch_adult(as_frame=True)
X = d.data.copy()
y = (d.target == ">50K").astype(int).to_numpy()
sex = X["sex"].astype(str).to_numpy()
Xtr_raw, Xte_raw, ytr, yte, gtr, gte = train_test_split(
    X, y, sex, test_size=0.3, random_state=0, stratify=y)

def fraction(num, den):
    return num / den if den else np.nan

def audit_adult(drop_sex=False):
    tr = Xtr_raw.drop(columns=["sex"]) if drop_sex else Xtr_raw.copy()
    te = Xte_raw.drop(columns=["sex"]) if drop_sex else Xte_raw.copy()
    cat = tr.select_dtypes(exclude=[np.number]).columns
    enc = OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1)
    train = tr.to_numpy(dtype=object); test = te.to_numpy(dtype=object)
    mask = np.array()
    train[:, mask] = enc.fit_transform(tr[cat].astype(str))
    test[:, mask] = enc.transform(te[cat].astype(str))
    m = HistGradientBoostingClassifier(categorical_features=mask, random_state=0)
    m.fit(train.astype(float), ytr)
    pred = m.predict_proba(test.astype(float))[:, 1] >= 0.5
    print("sex removed" if drop_sex else "all features")
    rates = []
    for group in ("Male", "Female"):
        k = gte == group; yy = yte[k]; pp = pred[k]
        tp = np.sum(pp & (yy == 1)); fp = np.sum(pp & (yy == 0))
        positives = np.sum(yy == 1); negatives = np.sum(yy == 0)
        row = [pp.mean(), fraction(tp, positives), fraction(fp, negatives), fraction(tp, pp.sum())]
        rates.append(row)
        print(f"{group} n={k.sum()} positives={positives} negatives={negatives} selected={pp.sum()}")
        print(f"base={yy.mean():.4f} sel={row[0]:.4f} TPR={row[1]:.4f} FPR={row[2]:.4f} PPV={row[3]:.4f}")
    print("absolute gaps: sel TPR FPR PPV", np.round(np.abs(np.array(rates[0])-rates[1]), 4))
    return rates

full_rates = audit_adult()
reduced_rates = audit_adult(drop_sex=True)
# all features
# Male n=9746 positives=2962 negatives=6784 selected=2519
# base=0.3039 sel=0.2585 TPR=0.6617 FPR=0.0824 PPV=0.7781
# Female n=4907 positives=544 negatives=4363 selected=390
# base=0.1109 sel=0.0795 TPR=0.5570 FPR=0.0199 PPV=0.7769
# absolute gaps: sel TPR FPR PPV [0.179  0.1047 0.0625 0.0012]
# sex removed
# Male n=9746 positives=2962 negatives=6784 selected=2504
# base=0.3039 sel=0.2569 TPR=0.6594 FPR=0.0812 PPV=0.7800
# Female n=4907 positives=544 negatives=4363 selected=397
# base=0.1109 sel=0.0809 TPR=0.5662 FPR=0.0204 PPV=0.7758
# absolute gaps: sel TPR FPR PPV [0.176  0.0932 0.0608 0.0041]

The two groups have different recorded income prevalence on the shared test rows. A group’s selection rate, TPR, FPR, and PPV describe different denominators, printed explicitly here. NaN would mean a rate is undefined because its denominator is zero; it should not be reported as zero disparity.

With all features, selection rates are 0.2585 and 0.0795, a gap of about 0.1790. The PPV gap is only 0.0012, while TPR and FPR gaps are 0.1047 and 0.0625. Similar precision among selected cases at this threshold does not establish calibration across the score range. Check score calibration separately if the outputs will be interpreted as probabilities. Differences in these rates depend on the chosen model, preprocessing, threshold, and evaluation population.

Removing sex reduces the selection gap from about 0.1790 to 0.1760 and the TPR gap from 0.1047 to 0.0932; the PPV gap increases from 0.0012 to 0.0041. Column removal changes the fitted model but does not require parity. Other features can carry group information, and groups can differ in label distributions and feature–label relationships. This two-model comparison does not establish which proxy the outcome model uses or a causal explanation for the remaining gap. The synthetic exercise below separately measures how well group membership can be predicted from the remaining features.

Excluding a group attribute from predictors and retaining it for evaluation are separate choices, as the code demonstrates. Decide how to collect, access, and retain such information in the context of the application and its obligations. A column-removal experiment alone cannot establish fairness or the appropriateness of collecting sensitive data.

Before choosing a mitigation, examine how features, labels, and missingness were produced. A selection gap can reflect a mix of model behavior, unequal access, historical decisions, and label measurement. Aggregate group averages can also conceal an intersectional subgroup with high error. For example, TPRs of 0.9 for 90 positives and 0.3 for 10 positives combine into 0.84; that average hides the second subgroup’s missed cases.

Counts determine how much precision the audit supports. A TPR of 8/10 and 800/1000 is 0.8 in both cases, but uncertainty is much larger in the first. Under independent sampling, use binomial intervals for fixed-model rates or resample evaluation rows to estimate uncertainty in gaps; repeated people require an appropriate clustered design. Report undefined rates and small groups, and account for exploratory subgroup searches before treating a discovered gap as a confirmed result.

Separate model fitting, threshold selection, and final evaluation. Compare mitigation candidates on validation data using relevant performance and harm measures, then assess the chosen procedure on untouched data. Reweighting changes training emphasis, constrained learning changes the objective, and threshold adjustment changes decisions after scoring. None guarantees that future-group rates will match validation rates. Revisit the audit when the population, labeling process, or decision policy changes.

Exercises

1. Deleting the protected attribute. Train with and without the group label and check what changes.

You should get: a gap that shrinks substantially and does not close, on features that still predict the group.

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, roc_auc_score

g = np.random.default_rng(0); n = 60_000
A = (g.random(n) < 0.5).astype(int)
x1 = 1.2 * A + g.normal(size=n)          # ordinary features that correlate with A
x2 = -0.8 * A + g.normal(size=n)
x3 = g.normal(size=n)
Y = (g.random(n) < 1 / (1 + np.exp(-(0.9 * x3 + 0.7 * A - 0.3)))).astype(int)

for name, X in (("includes A", np.c_[x1, x2, x3, A]), ("A removed", np.c_[x1, x2, x3])):
    Xtr, Xte, ytr, yte, Atr, Ate = train_test_split(X, Y, A, test_size=0.4, random_state=0)
    m = HistGradientBoostingClassifier(random_state=0).fit(Xtr, ytr)
    pred = (m.predict_proba(Xte)[:, 1] >= 0.5).astype(int)
    sel = [pred[Ate == a].mean() for a in (0, 1)]
    a_auc = roc_auc_score(Ate, HistGradientBoostingClassifier(random_state=0)
                          .fit(Xtr[:, :3], Atr).predict_proba(Xte[:, :3])[:, 1])
    print(f"{name:12s} accuracy {accuracy_score(yte, pred):.4f}"
          f"  selection g0 {sel[0]:.4f} g1 {sel[1]:.4f}  gap {abs(sel[0]-sel[1]):.4f}"
          f"   (A predictable from x1..x3 at AUC {a_auc:.4f})")
# includes A   accuracy 0.6738  selection g0 0.3487 g1 0.6872  gap 0.3384   (A predictable from x1..x3 at AUC 0.8407)
# A removed    accuracy 0.6675  selection g0 0.4511 g1 0.5678  gap 0.1167   (A predictable from x1..x3 at AUC 0.8407)

2. The price of the constraint. Trace the accuracy-parity frontier instead of picking a point on it.

You should get: a validation trade-off and held-out gaps that need not meet the validation caps exactly.

import numpy as np

g = np.random.default_rng(1); n = 200_000
A = (g.random(n) < 0.5).astype(int)
S = np.clip(np.where(A == 0, .30, .60) + .25*g.normal(size=n), .01, .99)
Y = (g.random(n) < S).astype(int)
valid = np.arange(n) < n//2
heldout = ~valid
grid = np.linspace(.05, .95, 91)

def threshold_stats(mask):
    selection, correct, sizes = [], [], []
    for a in (0, 1):
        k = mask & (A == a)
        decisions = S[k, None] >= grid[None, :]
        selection.append(decisions.mean(axis=0))
        correct.append((decisions == Y[k, None]).sum(axis=0))
        sizes.append(k.sum())
    accuracy = (correct[0][:, None] + correct[1][None, :]) / sum(sizes)
    gap = np.abs(selection[0][:, None] - selection[1][None, :])
    return accuracy, gap

va, vg = threshold_stats(valid)
ta, tg = threshold_stats(heldout)
base_i, base_j = np.unravel_index(np.argmax(va), va.shape)
print(f"unconstrained pair: {grid[base_i]:.2f}/{grid[base_j]:.2f} val={va[base_i,base_j]:.4f} test={ta[base_i,base_j]:.4f}")
print("allowed val-accuracy val-gap test-accuracy test-gap test-change thresholds")
for allowed in (.40, .30, .20, .10, .05, .02):
    feasible = vg <= allowed + 1e-12
    if not feasible.any():
        print(allowed, "no feasible grid pair")
        continue
    i, j = np.unravel_index(np.argmax(np.where(feasible, va, -np.inf)), va.shape)
    print(f"{allowed:.2f} {va[i,j]:.4f} {vg[i,j]:.4f} {ta[i,j]:.4f} {tg[i,j]:.4f} {ta[i,j]-ta[base_i,base_j]:+.4f} {grid[i]:.2f}/{grid[j]:.2f}")
# unconstrained pair: 0.49/0.52 val=0.7262 test=0.7245
# allowed val-accuracy val-gap test-accuracy test-gap test-change thresholds
# 0.40 0.7258 0.3930 0.7244 0.3905 -0.0001 0.48/0.52
# 0.30 0.7217 0.2969 0.7202 0.2928 -0.0043 0.44/0.55
# 0.20 0.7139 0.1997 0.7124 0.1972 -0.0120 0.44/0.61
# 0.10 0.7042 0.0965 0.7019 0.0916 -0.0225 0.40/0.64
# 0.05 0.6989 0.0489 0.6969 0.0474 -0.0275 0.37/0.64
# 0.02 0.6948 0.0174 0.6930 0.0168 -0.0314 0.35/0.64

3. Auditing against a label that was measured unequally. Give one group a higher detection rate and audit a model on the recorded label and on the truth.

You should get: different false-positive-rate gaps against recorded and true labels.

import numpy as np
from sklearn.linear_model import LogisticRegression

g = np.random.default_rng(0); n = 400_000
A = (g.random(n) < 0.5).astype(int)
x = g.normal(size=n)
p_true = 1 / (1 + np.exp(-(1.1 * x)))          # identical risk process in both groups
Y_true = (g.random(n) < p_true).astype(int)
detect = np.where(A == 1, 0.90, 0.55)          # group 1 is observed more thoroughly
Y_obs = Y_true * (g.random(n) < detect)

print(f"true prevalence      g0 {Y_true[A == 0].mean():.4f}   g1 {Y_true[A == 1].mean():.4f}")
print(f"RECORDED rate        g0 {Y_obs[A == 0].mean():.4f}   g1 {Y_obs[A == 1].mean():.4f}")

S = LogisticRegression(max_iter=2000).fit(x.reshape(-1, 1), Y_obs).predict_proba(x.reshape(-1, 1))[:, 1]
target = 0.60
th = {a: np.quantile(S[(A == a) & (Y_obs == 1)], 1 - target) for a in (0, 1)}
pred = np.where(A == 0, S >= th[0], S >= th[1]).astype(int)
print(f"thresholds tuned for equal TPR on the recorded label: {th[0]:.4f} / {th[1]:.4f}")

for label, Yv in (("RECORDED label", Y_obs), ("TRUE label", Y_true)):
    t = [pred[A == a][Yv[A == a] == 1].mean() for a in (0, 1)]
    f = [pred[A == a][Yv[A == a] == 0].mean() for a in (0, 1)]
    print(f"  audit vs {label:15s} TPR g0 {t[0]:.4f} g1 {t[1]:.4f} gap {abs(t[0]-t[1]):.4f}"
          f"   FPR g0 {f[0]:.4f} g1 {f[1]:.4f} gap {abs(f[0]-f[1]):.4f}")
# true prevalence      g0 0.5024   g1 0.4992
# RECORDED rate        g0 0.2753   g1 0.4495
# thresholds tuned for equal TPR on the recorded label: 0.3828 / 0.3827
#   audit vs RECORDED label  TPR g0 0.6000 g1 0.6000 gap 0.0000   FPR g0 0.3491 g1 0.2677 gap 0.0814
#   audit vs TRUE label      TPR g0 0.6000 g1 0.6004 gap 0.0004   FPR g0 0.2345 g1 0.2342 gap 0.0003

References

Chouldechova, Fair prediction with disparate impact; Fairlearn: Common fairness metrics; UCI Adult dataset.