\emph{Circuit analysis} is a promising technique for understanding the internal mechanisms of language models. However, existing analyses are done in small models far from the state of the art. To address this, we present a case study of circuit analysis in the 70B Chinchilla model, aiming to test the scalability of circuit analysis. In particular, we study multiple-choice question answering, and investigate Chinchilla's capability to identify the correct answer \emph{label} given knowledge of the correct answer \emph{text}. We find that the existing techniques of logit attribution, attention pattern visualization, and activation patching naturally scale to Chinchilla, allowing us to identify and categorize a small set of `output nodes' (attention heads and MLPs). We further study the `correct letter' category of attention heads aiming to understand the semantics of their features, with mixed results. For normal multiple-choice question answers, we significantly compress the query, key and value subspaces of the head without loss of performance when operating on the answer labels for multiple-choice questions, and we show that the query and key subspaces represent an `Nth item in an enumeration' feature to at least some extent. However, when we attempt to use this explanation to understand the heads' behaviour on a more general distribution including randomized answer labels, we find that it is only a partial explanation, suggesting there is more to learn about the operation of `correct letter' heads on multiple choice question answering.

翻译：电路分析是理解语言模型内部机制的一种有前景的技术。然而，现有分析均建立在远非最先进水平的小型模型上。为解决这一问题，我们针对700亿参数的Chinchilla模型开展电路分析案例研究，旨在检验电路分析的可扩展性。具体而言，我们研究选择题问答任务，探究Chinchilla在已知正确答案文本的前提下识别正确答案标签的能力。我们发现，对数概率归因、注意力模式可视化和激活修补等现有技术可自然扩展至Chinchilla模型，使我们能够识别并分类少量"输出节点"（注意力头与MLP层）。我们进一步研究"正确字母"类注意力头，试图理解其特征的语义，但结果参差不齐。针对常规选择题答案，我们在处理选择题答案标签时，可在不影响性能的前提下显著压缩注意力头的查询、键和值子空间，并证明查询和键子空间在一定程度上表征了"枚举中的第N项"特征。然而，当我们试图用这一解释来理解这些注意力头在包含随机答案标签的更广泛分布上的行为时，发现这仅能提供部分解释，表明我们对"正确字母"注意力头在选择题问答任务中的运作机制仍有待深入探索。