Can mechanistic interpretability scale to large models?

The skeptical view argues that scaling demonstrations and measurements is not the same as scaling trustworthy mechanistic understanding. Larger systems may contain more features and interactions than current tools can validate, while interventions can be fragile and architecture changes can invalidate transformer-focused assumptions. On this account, non-mechanistic behavioral prediction, control, or observational approaches may prove more practical for safety-critical use.

• A study of vision models found no interpretability improvement from increasing model or dataset scale; newer models could be less interpretable than older ones.
• Critical analyses argue that ablations and component isolation may misrepresent networks operating in context, favoring more observational approaches.
• SAE-based feature extraction and steering can reproduce basic results yet show fragility across layers, steering magnitudes, contexts, and thematically similar features.
• Critics therefore distinguish scalable inspection or feature discovery from globally valid causal understanding, warning that the former may not deliver reliable safety guarantees.
• Methods developed for standard transformers may face additional difficulties when applied to architectures such as state-space models, diffusion systems, and sparse mixture-of-experts models.

Featured sourceWhen the Coffee Feature Activates on Coffins: An Analysis of Feature Extraction and Steering for Mechanistic Interpretabilityarxiv.org7 sourcesView sources