Finding human-understandable circuits in language models is a central goal of the field of mechanistic interpretability. We train models to have more understandable circuits by constraining most of their weights to be zeros, so that each neuron only has a few connections. To recover fine-grained circuits underlying each of several hand-crafted tasks, we prune the models to isolate the part responsible for the task. These circuits often contain neurons and residual channels that correspond to natural concepts, with a small number of straightforwardly interpretable connections between them. We study how these models scale and find that making weights sparser trades off capability for interpretability, and scaling model size improves the capability-interpretability frontier. However, scaling sparse models beyond tens of millions of nonzero parameters while preserving interpretability remains a challenge. In addition to training weight-sparse models de novo, we show preliminary results suggesting our method can also be adapted to explain existing dense models. Our work produces circuits that achieve an unprecedented level of human understandability and validates them with considerable rigor.

翻译：在语言模型中寻找人类可理解的电路结构是机制可解释性领域的核心目标。我们通过约束模型的大部分权重为零来训练具有更易理解电路结构的模型，使得每个神经元仅具有少量连接。为了揭示多个手工设计任务背后的细粒度电路，我们对模型进行剪枝以隔离负责特定任务的部分。这些电路通常包含对应自然概念的神经元和残差通道，其间仅通过少量可直接解释的连接相互关联。我们研究了这些模型的缩放特性，发现使权重稀疏化是以能力换取可解释性，而扩大模型规模则能提升能力-可解释性边界。然而，在保持可解释性的前提下将稀疏模型的非零参数规模扩展至数千万以上仍面临挑战。除了从头训练权重稀疏模型外，我们的初步结果表明该方法也可用于解释现有的稠密模型。本研究构建的电路达到了前所未有的人类可理解水平，并通过严谨验证证明了其有效性。