Recent research in mechanistic interpretability has attempted to reverse-engineer Transformer models by carefully inspecting network weights and activations. However, these approaches require considerable manual effort and still fall short of providing complete, faithful descriptions of the underlying algorithms. In this work, we introduce a procedure for training Transformers that are mechanistically interpretable by design. We build on RASP [Weiss et al., 2021], a programming language that can be compiled into Transformer weights. Instead of compiling human-written programs into Transformers, we design a modified Transformer that can be trained using gradient-based optimization and then automatically converted into a discrete, human-readable program. We refer to these models as Transformer Programs. To validate our approach, we learn Transformer Programs for a variety of problems, including an in-context learning task, a suite of algorithmic problems (e.g. sorting, recognizing Dyck languages), and NLP tasks including named entity recognition and text classification. The Transformer Programs can automatically find reasonable solutions, performing on par with standard Transformers of comparable size; and, more importantly, they are easy to interpret. To demonstrate these advantages, we convert Transformers into Python programs and use off-the-shelf code analysis tools to debug model errors and identify the "circuits" used to solve different sub-problems. We hope that Transformer Programs open a new path toward the goal of intrinsically interpretable machine learning.

翻译：近期关于机制可解释性的研究试图通过仔细检查网络权重和激活来逆向工程Transformer模型。然而，这些方法需要大量人工操作，且仍未能提供底层算法的完整、忠实描述。在这项工作中，我们提出了一种训练Transformer的程序，使其在设计上具有机制可解释性。我们基于RASP [Weiss et al., 2021]——一种可编译为Transformer权重的编程语言——构建方法。不同于将人工编写的程序编译为Transformer，我们设计了一种改进的Transformer，可通过基于梯度的优化进行训练，随后自动转化为离散、人类可读的程序。我们将这些模型称为Transformer程序。为验证我们的方法，我们针对多种问题学习Transformer程序，包括上下文学习任务、一系列算法问题（如排序、Dyck语言识别）以及命名实体识别和文本分类等自然语言处理任务。Transformer程序能够自动找到合理解决方案，性能与同等规模的标准Transformer相当；更重要的是，它们易于解释。为展示这些优势，我们将Transformer转化为Python程序，并利用现成的代码分析工具调试模型错误，识别用于解决不同子问题的"电路"。我们期望Transformer程序为达成内在可解释机器学习目标开辟新路径。