SHAP (SHapley Additive exPlanations)

SHAP treats a prediction as a payout and the features as players who produced it. Each feature’s contribution is its Shapley value — its average marginal effect over every order in which the features could have been revealed. The contributions add up exactly:

\[ \hat{y} = \phi_0 + \sum_{i=1}^{M} \phi_i \]

where \(\phi_0\) is the model’s average output over a reference set and \(\phi_i\) is feature \(i\)’s share. Lundberg and Lee brought the construction from cooperative game theory, where Shapley proposed it in 1953, into machine learning as a unified account of additive feature attribution.

Shapley axioms describe a specified cooperative game: total attribution matches its payout, a player with no marginal contribution gets zero, symmetric players receive equal values, and values are additive across games. In SHAP, what counts as a missing feature defines the game. Interventional and conditional explanations can therefore allocate credit differently, especially when inputs are correlated.

Computing it exactly, and what falls out

import numpy as np
from itertools import combinations
from math import factorial
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(0)
n = 20_000
size = rng.normal(size=n)
rooms = 0.5 * size + rng.normal(scale=0.5, size=n)
noise_col = rng.normal(size=n)
price = 3.0 * size + rng.normal(scale=0.3, size=n)
"""'rooms' really does track price, but only through 'size'. 'noise_col' tracks
   nothing. The model is never shown 'rooms'."""

reference = rng.choice(n, 2000, replace=False)

def shapley(X, names, row_index=7):
    model = LinearRegression().fit(X, price)
    background = X[reference]
    row, M = X[row_index], X.shape[1]

    def value(subset):
        z = background.copy()
        for j in subset:
            z[:, j] = row[j]
        return model.predict(z).mean()

    phi = np.zeros(M)
    for j in range(M):
        others = [k for k in range(M) if k != j]
        for r in range(M):
            for subset in combinations(others, r):
                w = factorial(r) * factorial(M - r - 1) / factorial(M)
                phi[j] += w * (value(list(subset) + [j]) - value(list(subset)))
    parts = "  ".join(f"{nm} {v:+.4f}" for nm, v in zip(names, phi))
    print(f"  {parts}   base {value([]):+.4f}   base+sum {value([]) + phi.sum():+.4f}"
          f"   model says {model.predict(row.reshape(1, -1))[0]:+.4f}")

print("model given size and noise_col")
shapley(np.column_stack([size, noise_col]), ["size", "noise_col"])
print("the same model, with size supplied twice under two names")
shapley(np.column_stack([size, size, noise_col]), ["size", "floor_area", "noise_col"])
print(f"\n'rooms' was never given to either model. Its correlation with price: "
      f"{np.corrcoef(rooms, price)[0, 1]:.4f}")
# model given size and noise_col
#   size +2.7490  noise_col -0.0000   base +0.0919   base+sum +2.8409   model says +2.8409
# the same model, with size supplied twice under two names
#   size +1.3745  floor_area +1.3745  noise_col -0.0000   base +0.0919   base+sum +2.8409   model says +2.8409
#
# 'rooms' was never given to either model. Its correlation with price: 0.7017

Three readings from four lines.

The decomposition is exact. Base value plus contributions equals the model’s output, +2.8409, in both runs. This is not an approximation that happens to be close; the efficiency axiom makes it an identity. It is the property that lets you show someone a prediction and a complete account of where it came from.

In the second model defined by this code, the size signal is split symmetrically across two inputs. Their contributions are +1.3745 each instead of +2.7490 for the original single input, and the prediction remains +2.8409. That result depends on how the model uses the duplicated inputs and how missing inputs are represented; merely appending an unused duplicate column does not force a half-and-half split.

The symmetry axiom applies when the players have identical marginal contributions in the chosen game. Equal observed columns alone do not establish that property. Correlated features may share attribution under conditional SHAP even if the fitted function directly uses only one of them, while an interventional explanation can give an unused input zero. Report the explanation convention before interpreting a ranking of columns as a ranking of signals.

It explains the model, not the world

The last line of the run is the one people skip. rooms correlates with price at 0.7017. It is a real, useful signal about the outcome. It has no SHAP value, and never will, because the model was not given it.

Stated that way it sounds obvious. In practice it gets read backwards constantly: a feature absent from the SHAP summary is assumed not to matter, and a feature at the top is assumed to be what drives the outcome. Neither follows. SHAP answers “what did this model do with the inputs it was handed,” which is a narrower and more useful question than it is usually taken for.

Two corollaries worth stating to anyone reading a SHAP plot. A large value is not evidence of a causal effect — the model may be using a proxy, and the explanation will faithfully report the proxy, as the article on post-hoc explainability shows. And a zero is not evidence of irrelevance; it may mean the feature was withheld, or that a correlated column absorbed its role.

What the values are measured against

The reference distribution helps define the baseline and the attribution game. Changing it can change the base value and feature contributions, but need not change every contribution: an unused input under an interventional explanation, for example, can remain at zero. State the reference set so the values have a clear comparison population.

So “income contributed +0.2” is always shorthand for “+0.2 relative to this population.” Relative to applicants of the same age, or to last year’s applicants, or to approved applicants, the number can change — and which comparison is meant is usually the thing the person asking actually wants to know. State the reference set alongside the values.

Enumerating all 2^M subsets is expensive for an arbitrary black-box model. Production methods may approximate that calculation by sampling, while methods that exploit model structure can be exact: Tree SHAP provides polynomial-time algorithms under specified feature-dependence assumptions. Distinguish the chosen game from the algorithm used to compute or approximate its values, and check approximation uncertainty where applicable.

Using it well

• Group near-duplicate columns before ranking. Otherwise the ranking measures redundancy as much as importance.
• Report the reference set and the base value, not just the contributions.
• Do not use it to decide what to change. That is a question about interventions, and counterfactual explanations pose it directly — with their own caveats.
• Do not use it as a fairness check. A model can produce a wholly innocuous SHAP summary and still treat groups very differently, which is what fairness guardrails exist to catch.

Against LIME, SHAP’s advantage is that its values are defined rather than estimated from a local fit — there is a value for the specified game, which an implementation may compute exactly or approximate. That is a real difference, and it is not the same as the explanation being true of anything beyond the model.

References

• Lundberg & Lee, A Unified Approach to Interpreting Model Predictions, NeurIPS 2017
• SHAP documentation
• SHAP documentation, LinearExplainer and feature dependence
• SHAP documentation, TreeExplainer