\emph{Circuit analysis} is a promising technique for understanding the internal mechanisms of language models. However, existing analyses are done in small models far from the state of the art. To address this, we present a case study of circuit analysis in the 70B Chinchilla model, aiming to test the scalability of circuit analysis. In particular, we study multiple-choice question answering, and investigate Chinchilla's capability to identify the correct answer \emph{label} given knowledge of the correct answer \emph{text}. We find that the existing techniques of logit attribution, attention pattern visualization, and activation patching naturally scale to Chinchilla, allowing us to identify and categorize a small set of `output nodes' (attention heads and MLPs). We further study the `correct letter' category of attention heads aiming to understand the semantics of their features, with mixed results. For normal multiple-choice question answers, we significantly compress the query, key and value subspaces of the head without loss of performance when operating on the answer labels for multiple-choice questions, and we show that the query and key subspaces represent an `Nth item in an enumeration' feature to at least some extent. However, when we attempt to use this explanation to understand the heads' behaviour on a more general distribution including randomized answer labels, we find that it is only a partial explanation, suggesting there is more to learn about the operation of `correct letter' heads on multiple choice question answering.

翻译：\textit{电路分析}是一种理解语言模型内部机制的前沿技术，但现有分析多局限于远离当前最优水平的小型模型。为解决此问题，我们针对700亿参数的Chinchilla模型开展电路分析案例研究，旨在检验电路分析在大规模模型中的可扩展性。具体而言，我们研究多项选择问答任务，探讨Chinchilla在已知正确答案文本的情况下识别正确答案\textit{标签}的能力。研究发现，logit归因、注意力模式可视化与激活修补等现有技术可自然扩展到Chinchilla模型，使我们能够识别并分类少量“输出节点”（注意力头与MLP）。我们进一步研究“正确字母”类别的注意力头，尝试理解其语义特征，但结果呈现混合性。对于常规多项选择答案，当处理多项选择问题的答案标签时，我们能在不损失性能的情况下显著压缩注意力头的查询、键和值子空间，并证明其查询与键子空间至少在某种程度上表征了“枚举中的第N项”特征。然而，当我们试图用这一解释来理解注意力头在包含随机答案标签的泛化分布上的行为时，发现这仅是部分解释，表明关于“正确字母”注意力头在多项选择问答中的运作机制仍有待深入探究。