Interpretability methods aim to understand the algorithm implemented by a trained model (e.g., a Transofmer) by examining various aspects of the model, such as the weight matrices or the attention patterns. In this work, through a combination of theoretical results and carefully controlled experiments on synthetic data, we take a critical view of methods that exclusively focus on individual parts of the model, rather than consider the network as a whole. We consider a simple synthetic setup of learning a (bounded) Dyck language. Theoretically, we show that the set of models that (exactly or approximately) solve this task satisfy a structural characterization derived from ideas in formal languages (the pumping lemma). We use this characterization to show that the set of optima is qualitatively rich; in particular, the attention pattern of a single layer can be ``nearly randomized'', while preserving the functionality of the network. We also show via extensive experiments that these constructions are not merely a theoretical artifact: even after severely constraining the architecture of the model, vastly different solutions can be reached via standard training. Thus, interpretability claims based on inspecting individual heads or weight matrices in the Transformer can be misleading.

翻译：可解释性方法旨在通过检查模型的各个方面（如权重矩阵或注意力模式）来理解已训练模型（如Transformer）所实现的算法。本文通过理论结果与合成数据上精心控制的实验相结合，对仅关注模型局部而非整体网络的现有方法持批判态度。我们考虑学习（有界）Dyck语言这一简单合成任务。理论上，我们证明求解（精确或近似）该任务的模型集合满足源于形式语言理论（泵引理）的结构特征。利用该特征可揭示最优解集合在定性上的丰富性：尤其当保持网络功能不变时，单层注意力模式可能呈现"近乎随机"的状态。通过大量实验进一步证实，这些构造并非纯粹的理论产物：即使对模型架构施加严格约束，标准训练仍能收敛到截然不同的解。因此，基于检查Transformer中单个注意力头或权重矩阵的可解释性结论可能具有误导性。