Post-hoc Explainability

Post-hoc explainability means analyzing a model after it has been trained, without altering it. It is the alternative to intrinsic interpretability, where a linear model or a small decision tree is legible by construction and you give up whatever accuracy that costs.

The toolbox is broad: global and local feature importance, surrogate models that approximate the black box with something simpler, LIME and SHAP for individual predictions, partial dependence and ICE plots for marginal effects, saliency maps and Grad-CAM for images, and counterfactuals for “what would have to change.”

The standard warning about all of them is that an explanation may be unfaithful — an approximation that does not match what the model really does. That is true and worth guarding against. The harder case is the opposite one.

A faithful importance estimate of a problem it cannot show you

The following synthetic example makes a postcode-related feature correlate with group membership and uses it to generate recorded approvals. It illustrates why feature importance and group outcome checks answer different questions.

import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
n = 20_000
group = rng.integers(0, 2, size=n)
postcode_income = rng.normal(loc=np.where(group == 0, -0.9, 0.9), scale=0.7, size=n)
history = rng.normal(size=n)
payments = rng.normal(size=n)
"""The protected attribute is never given to the model. 'postcode_income' is not a
   protected attribute either - but it tracks one, and the recorded decisions used it."""
approve = (rng.random(n) < 1 / (1 + np.exp(-(1.2 * history + 0.8 * payments
                                             + 1.5 * postcode_income)))).astype(int)

names = ["postcode_income", "history", "payments"]
X = np.column_stack([postcode_income, history, payments])
xtr, xte, ytr, yte, gtr, gte = train_test_split(X, approve, group, test_size=0.3, random_state=0)
model = GradientBoostingClassifier(random_state=0).fit(xtr, ytr)

imp = permutation_importance(model, xte, yte, n_repeats=5, random_state=0)
print("what the explanation reports")
for name, m in zip(names, imp.importances_mean):
    print(f"  permutation importance  {name:<16} {m:+.4f}")

p = model.predict_proba(xte)[:, 1]
print("\nwhat the model does, by a group it was never shown")
for g in (0, 1):
    m = gte == g
    print(f"  group {g}   mean predicted approval {p[m].mean():.4f}"
          f"   selected at 0.5 {np.mean(p[m] >= 0.5):.4f}   actual approval rate {yte[m].mean():.4f}")
print(f"\ncorrelation between postcode_income and the protected attribute: "
      f"{np.corrcoef(postcode_income, group)[0, 1]:.4f}")
# what the explanation reports
#   permutation importance  postcode_income  +0.1921
#   permutation importance  history          +0.0918
#   permutation importance  payments         +0.0395
#
# what the model does, by a group it was never shown
#   group 0   mean predicted approval 0.3064   selected at 0.5 0.2502   actual approval rate 0.2995
#   group 1   mean predicted approval 0.7105   selected at 0.5 0.7823   actual approval rate 0.7170
#
# correlation between postcode_income and the protected attribute: 0.7886

The model approves group 1 at 78.23% and group 0 at 25.02%. It was never given the protected attribute; there is no line of code to point at.

And the explanation is correct. postcode_income really is the most important feature, at +0.1921 permutation importance against +0.0918 for credit history. Anyone reading that output learns something true about the model. What they do not learn is that the feature correlates with the protected attribute at 0.7886, that the recorded approvals it was fitted on already reflected that, and that the model reproduces a threefold gap.

Nothing failed here. The explanation answered the question it was asked — how much shuffling each input reduces this model’s predictive accuracy on the evaluation data — and that question has no group in it. A review that accepts “we ran SHAP and nothing looked like a protected attribute” as evidence of fairness has confused two different checks.

The check that would find it is a direct one: compute the outcome rates and error rates by group, whether or not the model was given the group. That is what fairness guardrails do, and no amount of feature attribution substitutes for it.

What post-hoc methods are actually for

• Debugging. This is where they are strongest and least disputed. An importance ranking that puts a leaked identifier or a timestamp at the top has found a real bug, quickly.
• Sanity-checking against domain knowledge. If the clinicians say the third feature should not matter and it dominates, someone has learned something — either about the model or about the clinicians.
• Giving an affected person something to act on. Wachter, Mittelstadt and Russell make the useful distinction: an explanation can aim to help someone act rather than to make them understand the system, and that kind can be “offered without opening the black box.”
• Documentation. A record of what the model attended to at release, which later drift can be compared against.

What they are not for is establishing that a model is causally correct, or safe, or fair. Those are claims about the world and about outcomes; post-hoc methods make claims about a function.

Reading an explanation carefully

• Ask what the method was faithful to. A surrogate model’s fidelity to the black box is a number you can compute; if nobody reports it, the explanation is unaudited.
• Ask what the feature stands for. A postcode is a postcode until you ask what else it predicts. Correlations between “neutral” features and protected attributes belong in the same report.
• Ask what is missing. Features the model was not given have no attribution, however informative they are — the article on SHAP shows one correlated at 0.7017 with the outcome and absent from the explanation entirely.
• Ask whether it is stable. Rerun it. Methods that sample can give different answers to the same question.

And where the stakes justify it, the alternative remains available: a model that is interpretable by construction needs no explanation layer, and the accuracy gap is often smaller than assumed. Post-hoc explanation is what you do when you have chosen not to take that route — a reasonable choice, made more safely when its limits are stated.

References

• Wachter, Mittelstadt & Russell, Counterfactual Explanations without Opening the Black Box, Harvard Journal of Law & Technology 31(2), 2018
• scikit-learn User Guide, Permutation feature importance
• Lundberg & Lee, A Unified Approach to Interpreting Model Predictions, NeurIPS 2017