LIME

LIME explains one prediction at a time. Rather than describing the whole model, it builds a small dataset around the instance in question, asks the model what it thinks of each point, weights those points by how close they are to the original, and fits a simple model — usually a linear one — to that weighted neighborhood. The simple model’s coefficients are the explanation.

Ribeiro, Singh and Guestrin introduced it as a way to explain any classifier “in an interpretable and faithful manner, by learning an interpretable model locally around the prediction.” It works on text, images and tables, and it needs nothing but the ability to call the model.

The procedure has two knobs that nobody hands you a value for, and between them they determine the answer.

The two settings, measured

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import Ridge

rng = np.random.default_rng(0)
n, d = 4000, 6
X = rng.normal(size=(n, d))
logit = 2.0 * X[:, 0] - 1.5 * X[:, 1] + 1.0 * X[:, 0] * X[:, 2]
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
model = RandomForestClassifier(n_estimators=200, random_state=0).fit(X, y)
target = X[3]

def lime(seed, samples=5000, width=1.0):
    """LIME's recipe: sample around the instance, weight by proximity,
       fit a weighted linear model, report its coefficients and its own fit."""
    local = np.random.default_rng(seed)
    z = target + local.normal(size=(samples, len(target)))
    w = np.exp(-((z - target) ** 2).sum(axis=1) / width ** 2)
    p = model.predict_proba(z)[:, 1]
    fit = Ridge(alpha=1.0).fit(z, p, sample_weight=w)
    return fit.coef_, fit.score(z, p, sample_weight=w)

print(f"the model predicts {model.predict_proba(target.reshape(1, -1))[0, 1]:.4f} for this instance\n")
print("how many perturbations, and which feature comes out on top over five seeds")
for samples in (100, 500, 2000, 10000):
    runs = [lime(s, samples=samples) for s in range(1, 6)]
    c = np.array([r[0] for r in runs])
    print(f"  {samples:6d} samples   x0 ranges {c[:, 0].min():+.4f} to {c[:, 0].max():+.4f}"
          f"   x1 ranges {c[:, 1].min():+.4f} to {c[:, 1].max():+.4f}"
          f"   top feature per seed {[int(np.abs(r[0]).argmax()) for r in runs]}")

print("\nthe kernel width, at 5000 samples and one fixed seed")
for width in (0.25, 0.5, 1.0, 2.0, 4.0):
    coef, r2 = lime(1, samples=5000, width=width)
    print(f"  width {width:4.2f}   " + "  ".join(f"{v:+.4f}" for v in coef)
          + f"   the surrogate's own R2 {r2:.4f}")
# the model predicts 0.7250 for this instance
#
# how many perturbations, and which feature comes out on top over five seeds
#      100 samples   x0 ranges +0.1157 to +0.1986   x1 ranges -0.1538 to -0.0897   top feature per seed [0, 1, 1, 0, 0]
#      500 samples   x0 ranges +0.2728 to +0.2805   x1 ranges -0.2222 to -0.1907   top feature per seed [0, 0, 0, 0, 0]
#     2000 samples   x0 ranges +0.3159 to +0.3236   x1 ranges -0.2479 to -0.2256   top feature per seed [0, 0, 0, 0, 0]
#    10000 samples   x0 ranges +0.3268 to +0.3315   x1 ranges -0.2514 to -0.2370   top feature per seed [0, 0, 0, 0, 0]
#
# the kernel width, at 5000 samples and one fixed seed
#   width 0.25   +0.0019  -0.0020  +0.0007  +0.0024  +0.0010  -0.0042   the surrogate's own R2 0.0239
#   width 0.50   +0.1645  -0.1625  +0.0277  +0.0209  +0.0295  +0.0034   the surrogate's own R2 0.5045
#   width 1.00   +0.3304  -0.2394  +0.0580  -0.0036  +0.0191  +0.0292   the surrogate's own R2 0.8010
#   width 2.00   +0.2543  -0.1751  +0.0483  -0.0022  +0.0057  +0.0180   the surrogate's own R2 0.7849
#   width 4.00   +0.2192  -0.1549  +0.0424  -0.0011  +0.0050  +0.0142   the surrogate's own R2 0.7715

This is LIME’s recipe implemented directly rather than the published package, so the discretization and feature selection that lime adds are not here. The two sensitivities below are properties of the recipe, not artifacts of one implementation.

How many perturbations. At 100 samples, five runs that differ only in their random seed pick x0, x1, x1, x0, x0 as the most important feature. The headline sentence of the explanation — “the main reason was your income” versus “the main reason was your debt” — depends on the draw. By 500 samples the ranking is stable and by 2000 the coefficients agree to two decimals.

In this experiment, more perturbations stabilized the top feature across the five seeds. Repeat the explanation with different seeds and assess stability within the chosen neighborhood. Increasing the sample count can reduce sampling variability, but it does not validate the neighborhood, perturbation distribution or surrogate specification.

How wide the neighborhood. Changing the width changes the local question; more samples alone do not establish which neighborhood is appropriate. At width 0.25 every coefficient is within 0.005 of zero and the surrogate’s own \(R^2\) is 0.0239 — the linear fit explains essentially nothing about the model’s behavior in that region, but this result alone does not identify the cause. Narrow kernels reduce the total weight and effective sample support, while the Ridge penalty stays fixed at 1.0. Sampling, regularization and local model behavior all need checking. At width 1.00 the coefficients are roughly 1.5 times their values at width 4.00.

Every one of those runs returns a ranked list of features that looks like an explanation. The low local fit warns that the 0.25 version does not reproduce the model well under these weights; it does not by itself prove that noise is the cause.

Read the surrogate’s fit, always

The \(R^2\) column is the part of LIME’s output that answers “should I believe this.” It says how well a linear model reproduces the black box in that neighborhood. When it is high, the coefficients are a fair description of what the model does nearby. When it is low, the penalized surrogate has not reproduced the sampled, weighted predictions well — which is a different object, and not one to hand to a customer.

A low value is informative rather than fatal. It can reflect non-linear model behavior, sparse effective support or regularization. Investigate these possibilities before attributing it to an interaction or threshold. Report it alongside the coefficients.

Against SHAP

The comparison is usually made as “LIME is unstable, SHAP is theoretically grounded,” which is close enough to be misleading. The theory behind SHAP concerns the definition of the values — Lundberg and Lee show there is a unique attribution in their class with a set of desirable properties. It is not a statement about estimator variance, and KernelSHAP samples too.

The real difference is what the number is. A SHAP value is defined; implementations approximate it, and a better implementation gets closer to a fixed target. A LIME coefficient is the output of a fitting procedure — change the neighborhood and you have not approximated the answer better or worse, you have asked a different question. There is no width that is correct.

What LIME offers in exchange is cheapness and reach. It needs a handful of model calls rather than an enumeration, it works on any input representation, and its output — a weighted list of human-scale features — is what a non-specialist can actually read.

Using it

• Repeat with several seeds before showing anyone the result. If the ranking moves, the explanation is not yet an explanation.
• Report the kernel width and the local fit. Two teams running LIME on the same instance with different widths will disagree, and without those numbers the disagreement is unresolvable.
• Watch the truncation. Asking for the top \(k\) features hides the rest, and if the model is spreading credit across correlated columns the top \(k\) may be an arbitrary slice.
• Remember what it is faithful to. Like every post-hoc method, LIME describes the model’s behavior, which may rest on a proxy rather than on anything causal.

For the question “what would I have to change,” a local linear coefficient is a poor guide anyway — extrapolating it far from the instance leaves the neighborhood it was fitted in. Counterfactual explanations ask that question directly.

References

• Ribeiro, Singh & Guestrin, “Why Should I Trust You?”: Explaining the Predictions of Any Classifier, KDD 2016
• Lundberg & Lee, A Unified Approach to Interpreting Model Predictions, NeurIPS 2017