A prominent research direction in mechanistic interpretability is learning sparse circuits over LLM components to reveal how they jointly produce model behavior. However, raw neurons are polysemantic, making learned circuits hard to interpret. Sparse autoencoder (SAE) features alleviate this, but their high dimensionality makes existing intervention-based circuit learning methods computationally prohibitive. We propose CircuitLasso, a scalable circuit-learning approach based on sparse linear regression. CircuitLasso recovers circuits whose structural accuracy matches that of state-of-the-art intervention-based methods on the benchmark data, at a fraction of the computational cost. For interpretability, CircuitLasso efficiently uncovers relationships among SAE features, showing how human-interpretable semantic features propagate through the model and influence its predictions. Finally, we validate the utility of our learned circuits by leveraging their insights to achieve comparable performance at substantially lower cost on a domain-generalization task.

翻译：在机械可解释性领域，一个重要研究方向是通过学习LLM组件上的稀疏电路来揭示它们如何共同产生模型行为。然而，原始神经元具有多语义性，使得学习到的电路难以解释。稀疏自编码器（SAE）特征缓解了这一问题，但其高维度特性使得现有基于干预的电路学习方法在计算上难以承受。我们提出CircuitLasso，一种基于稀疏线性回归的可扩展电路学习方法。在基准数据集上，CircuitLasso恢复的电路结构精度与最先进的基于干预的方法相当，而计算成本仅为后者的一小部分。为提升可解释性，CircuitLasso能高效揭示SAE特征间的关系，展示人类可解释的语义特征如何在模型中传播并影响其预测。最后，我们通过利用所学电路洞察力，在领域泛化任务中以显著更低的成本实现可比较的性能，验证了这些电路的实际效用。