Do post-hoc explanations mislead?

Lens adapted to this topic: Stronger claims that post-hoc explanations are misleading

A skeptical reading holds that post-hoc explanations often rationalize outputs rather than reveal the decision process. Because they can be ambiguous, manipulable, vulnerable to spurious correlations, and over-interpreted by users, they may create false confidence—particularly in adversarial or high-stakes settings.

• LIME and SHAP can exhibit explanatory inversion: explanations may rely on the output and depart from the model’s true input-output relationship, especially with spurious correlations.
• Feature-additive explainers can misattribute importance when models contain feature interactions, even when evaluated against known ground truth.
• Post-hoc explanations were found ineffective for detecting unknown spurious signals and could falsely indicate reliance on them.
• In adversarial settings, explanation ambiguity can allow providers to manipulate explanations, undermining their suitability for accountability purposes.
• Explanations can be deliberately constructed or selected in ways that manipulate expert users’ trust in problematic black-box models.

Featured sourceAre We Merely Justifying Results ex Post Facto? Quantifying Explanatory Inversion in Post-Hoc Model Explanationsarxiv.org5 sourcesView sources