1) Definition

  • Post-hoc explainability = explaining a machine learning model’s behavior after training, without changing its internal structure.
  • Focuses on making black-box models (like deep neural networks, ensembles, gradient boosting) more interpretable.
  • Contrast with intrinsically interpretable models (linear regression, small decision trees), which are explainable by design.

“Post-hoc” = after the fact. You don’t train the model to be interpretable; you analyze its outputs afterward.

2) Why Post-hoc Explainability?

  • Many high-performing models (deep learning, boosted trees) are opaque.
  • Stakeholders (users, regulators, business) need to know why a prediction was made.
  • Used for:
    • Debugging models
    • Trust & transparency for users
    • Compliance (finance, healthcare, GDPR/AI Act)

3) Common Techniques

a) Feature importance

  • Global: Which features drive predictions overall?
    • Example: permutation importance, gain-based importance in tree models.
  • Local: Which features influenced a specific prediction?

b) Surrogate models

  • Train a simpler, interpretable model (e.g., decision tree) to approximate the black-box model.

c) Local explanation methods

  • LIME (Local Interpretable Model-agnostic Explanations):
    • Perturb inputs, fit a local linear model around the instance.
  • SHAP (SHapley Additive exPlanations):
    • Based on cooperative game theory; assigns contribution values to each feature.

d) Visualization techniques

  • Partial Dependence Plots (PDPs): show marginal effect of one feature on predictions.
  • ICE plots (Individual Conditional Expectation): show instance-level variation.
  • Saliency maps / Grad-CAM (for deep learning): highlight important pixels in images.

e) Counterfactual explanations

  • Answer the question: “What minimal change would flip the prediction?”
    • Example: “If income was +$5,000, loan approval would change from denied → approved.”

4) Examples

  • Loan model (black box): Predicts denied.
    • SHAP: Top contributors = low income, short employment history.
    • Counterfactual: If employment history was +2 years, approval would flip.
  • Image classifier: Predicts “cat.”
    • Saliency map shows highlighted ears/whiskers region → helps validate the model is looking at the right features.

5) Limitations

  • Post-hoc methods may be approximations of the real model logic.
  • Risk of misleading explanations (faithfulness problem).
  • Computationally expensive (e.g., SHAP on large datasets).
  • Not a substitute for fair training practices (only diagnostic).

6) Summary

  • Post-hoc explainability = making opaque ML models interpretable after training.
  • Techniques: feature importance, LIME, SHAP, PDPs, counterfactuals, saliency maps.
  • Useful for trust, debugging, compliance.
  • Trade-off: explanations approximate, may not fully reveal model internals.