Model interpretability

Lens adapted to this topic: The case for useful, carefully evaluated interpretations

The mainstream technical view treats interpretability as a context-dependent property: an explanation is useful when a particular audience can understand or predict model behavior well enough for its purpose. It supports a range of inherently interpretable and post-hoc methods, while emphasizing that methods should be evaluated rather than accepted merely because they produce persuasive visualizations. Applications include debugging, assessing subgroup behavior, supporting human oversight, and meeting governance needs.

• Interpretability is audience- and purpose-dependent: an interpretation should help people understand causes or predict a model’s results efficiently and correctly.
• Interpretability research includes inherently interpretable models as well as post-hoc methods such as feature attribution and other model-agnostic techniques.
• Interpretation methods should be assessed using criteria such as predictive accuracy, descriptive accuracy, and relevance, rather than appearance alone.
• Global and local explanations can support debugging, fairness analysis, human-AI collaboration, and regulatory assessment.

Lens adapted to this topic: Critiques of interpretability and post-hoc explanations

Skeptical researchers argue that “interpretability” is too ambiguous to function as a reliable technical target and that many XAI techniques lack clear purposes or strong validation. They question whether feature-attribution graphics and other post-hoc outputs reveal a model’s actual causes or reasons. A stronger version of this critique holds that, especially in high-stakes settings, explanations of black boxes may create false reassurance and should not substitute for inherently interpretable models.

• Because interpretability has multiple unclear meanings, technical features cannot automatically be treated as satisfying the concept or solving the problems motivating it.
• Some XAI work uses attractive heatmaps or weak benchmarks without adequately establishing what the explanations mean or whether they are reliable.
• Post-hoc XAI outputs may not qualify as reason explanations in the causal, dispositional, or normative sense associated with explaining why an agent decided something.
• For high-stakes decisions, critics argue that explaining black-box models can perpetuate harmful practice and that inherently interpretable models are preferable where feasible.
• Skeptical discussions also challenge whether commonly used explanation techniques actually reveal the internal mechanisms responsible for a model’s behavior.