Counterfactual Explanations

A counterfactual explanation answers a different question from a feature attribution. Instead of “which inputs drove this decision,” it asks “what would have to be different for the decision to change.” Formally, given an instance \(x\) with an unwanted prediction, find

\[ \min_{x’} \; d(x, x’) \quad \text{subject to} \quad f(x’) = y_{\text{desired}} \]

for some distance \(d\). Wachter, Mittelstadt and Russell proposed it in the context of the GDPR’s right to explanation, on the argument that an explanation can aim “to help a data subject act rather than merely understand,” and that such an explanation can be given “without opening the black box.”

The appeal is obvious — “earn \$5,000 more and you would be approved” is the sentence a rejected applicant actually wants. Two things about that sentence need care.

There is no such thing as the counterfactual

import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
n = 20_000
income = rng.normal(48_000, 14_000, size=n)
years_employed = np.clip(rng.normal(6, 4, size=n), 0, None)
open_accounts = rng.integers(1, 9, size=n).astype(float)
age = rng.normal(41, 11, size=n)
"""Age is in the model because it was in the data. Whether it should be is a
   separate question - but a counterfactual search will happily use it."""

names = ["income", "years_employed", "open_accounts", "age"]
units = ["dollars", "years", "accounts", "years"]
X = np.column_stack([income, years_employed, open_accounts, age])
approve = (rng.random(n) < 1 / (1 + np.exp(-((income - 48_000) / 12_000
                                             + 0.25 * years_employed
                                             - 0.30 * open_accounts
                                             + 0.04 * (age - 41) - 0.5)))).astype(int)
model = LogisticRegression(max_iter=5000).fit(X / X.std(axis=0), approve)
scale = X.std(axis=0)
applicant = np.array([39_000.0, 2.0, 6.0, 29.0])

def decision(row):
    return model.predict_proba((row / scale).reshape(1, -1))[0, 1]

print(f"the applicant is denied: predicted approval {decision(applicant):.4f}\n")
print("the smallest single-feature change that flips the decision to 0.50")
for j, (name, unit) in enumerate(zip(names, units)):
    step = scale[j] / 2000
    for direction in (1, -1):
        row, moves = applicant.copy(), 0
        while decision(row) < 0.5 and moves < 40_000:
            row[j] += direction * step
            moves += 1
        if decision(row) >= 0.5:
            change = row[j] - applicant[j]
            print(f"  {name:<15} {applicant[j]:>10.1f} -> {row[j]:>10.1f}"
                  f"   a change of {change:+.1f} {unit}"
                  f"   ({abs(change) / scale[j]:.2f} standard deviations)")
            break
# the applicant is denied: predicted approval 0.0491
#
# the smallest single-feature change that flips the decision to 0.50
#   income             39000.0 ->    75527.7   a change of +36527.7 dollars   (2.62 standard deviations)
#   years_employed         2.0 ->       14.1   a change of +12.1 years   (3.18 standard deviations)
#   open_accounts          6.0 ->       -3.6   a change of -9.6 accounts   (4.20 standard deviations)
#   age                   29.0 ->      108.0   a change of +79.0 years   (7.21 standard deviations)

This search moves one feature at a time, so a real tool would find closer solutions that change several at once. The two properties on display survive that: the answer is not unique and nothing keeps it plausible.

All four rows are valid counterfactuals. Each flips the decision, each approximates the smallest change along its own axis to within one search step, and they recommend incompatible things. Which one is “closest” is entirely a matter of \(d\): in standard deviations income wins at 2.62, but a distance that weighted features by how hard they are to change — money being easier to acquire than a decade of tenure — would order them differently, and a distance in raw units would be meaningless across dollars, years and counts.

So the metric is not a technical detail to be defaulted. It is the thing that decides what the system tells the applicant to do, and it encodes a judgment about what is easy for them — a judgment usually made by whoever wrote the code, about someone they have never met.

The last two rows are worse. Minus 3.6 open accounts does not exist. An age of 108 is not a plan. The optimizer returns them because the model is a function defined over the whole space, and that space contains regions no applicant occupies. Generating impossible counterfactuals is not an edge case to be guarded against; without explicit constraints on ranges, types and immutability, it is the default behavior.

The age row also shows why immutability constraints are not optional in a different sense: a model that would approve the applicant at 108 is using age, and telling someone their loan depends on their age is a statement with legal consequences that a distance metric arrived at by accident.

It is a fact about the model, not about the world

“If income were \$36,528 higher, the model would approve” is a true statement about a function. It is not a prediction of what happens if the applicant earns more.

Editing one cell of the input holds every other cell fixed. Actually earning more changes things that travel with income — the debt ratio shifts, the account profile changes, time passes and tenure grows. The model would then be evaluated at a different point than the counterfactual named, and its answer there is unknown. The gap between “change the input” and “change the world” is the gap between a sensitivity analysis and a causal claim, and only the first is on offer.

This matters most where the explanation is given as advice. A counterfactual delivered to a customer is read as a promise: do this and you will be approved. What it supports is narrower — under this model, unchanged, an application identical to yours except in this field would have been approved. Whether that is honest enough to hand over depends on how stable the model is and how correlated the feature is with everything else.

What makes one usable

• Valid — it actually flips the decision, checked against the deployed model rather than a local approximation.
• Plausible — it lands where real records land. Range and type constraints are the minimum; a density check against the training data is better.
• Actionable — it only moves features the person can move. Age, place of birth and any protected characteristic are fixed inputs to the search, not candidates.
• Sparse — few enough changes to state in a sentence.
• Diverse — several options rather than one, precisely because the metric’s choice among them is arbitrary. Showing three lets the person pick the one that is cheap for them.

Used as an audit tool rather than as advice, counterfactuals are sharper: if members of one group systematically need larger changes to reach the same decision, that is a disparity worth investigating, and it is a question about outcomes rather than about features — the kind that fairness guardrails are built to ask.

Alongside the attribution methods, the division of labor is clean. SHAP apportions credit for the decision that was made; LIME approximates the model near the instance; a counterfactual names a point where the decision is different. All three are post-hoc, and all three describe the model.

References

• Wachter, Mittelstadt & Russell, Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR, Harvard Journal of Law & Technology 31(2), 2018
• DiCE (Diverse Counterfactual Explanations) documentation