Circuit discovery, which involves identifying sparse and task-relevant subnetworks in pre-trained language models, is a cornerstone of mechanistic interpretability. Automated Circuit Discovery (ACDC) has emerged as a pivotal methodology in circuit discovery, but its application to large language models is severely limited by computational inefficiency and prohibitively high memory requirements. Although several accelerated approaches have been proposed, they primarily rely on linear approximations to ACDC, which significantly compromises analytical faithfulness. Our proposed method for accelerating automated circuit discovery, Per Attention Head Quantization (PAHQ), takes a fundamentally different approach by optimizing the efficiency of each individual patching operation. PAHQ leverages a fundamental alignment between activation patching and mixed-precision quantization (MPQ): interpretability analysis through patching essentially performs targeted ablation studies. Therefore, we can maintain high precision exclusively for investigated components while safely reducing precision elsewhere in the network. PAHQ-accelerated ACDC reduces runtime by up to 80\% and memory consumption by up to 30\% compared to unaccelerated ACDC while maintaining faithfulness. Importantly, our method readily integrates with existing edge-based circuit discovery techniques by modifying the attention computation mechanism. This training-free approach provides a practical and novel pathway for accelerating mechanistic interpretability methods. Our code is available at https://github.com/626619403/PAHQ.

翻译：电路发现是机械可解释性研究的基石，其目标在于识别预训练语言模型中稀疏且与任务相关的子网络。自动化电路发现（ACDC）已成为电路发现的关键方法，但其在大型语言模型中的应用受限于计算效率低下和极高的内存需求。尽管已有若干加速方案被提出，这些方法主要依赖于对ACDC的线性近似，这严重损害了分析保真度。我们提出的加速自动化电路发现方法——注意力头量化（PAHQ）采用根本不同的优化路径，通过优化每个独立修补操作的效率实现加速。PAHQ利用了激活修补与混合精度量化（MPQ）之间的本质关联：通过修补进行的可解释性分析本质上执行的是定向消融研究。因此，我们可以仅对研究组件保持高精度，同时在网络其他部分安全地降低精度。相较于未加速的ACDC，PAHQ加速的ACDC在保持保真度的同时，将运行时间减少高达80%，内存消耗降低高达30%。重要的是，我们的方法通过修改注意力计算机制，能够与现有的基于边缘的电路发现技术无缝集成。这种免训练的方法为加速机械可解释性方法提供了实用且新颖的路径。我们的代码公开于 https://github.com/626619403/PAHQ。