Following their success across many domains, transformers have also proven effective for symbolic regression (SR); however, the internal mechanisms underlying their generation of mathematical operators remain largely unexplored. Although mechanistic interpretability has successfully identified circuits in language and vision models, it has not yet been applied to SR. In this article, we introduce PATCHES, an evolutionary circuit discovery algorithm that identifies compact and correct circuits for SR. Using PATCHES, we isolate 28 circuits, providing the first circuit-level characterisation of an SR transformer. We validate these findings through a robust causal evaluation framework based on key notions such as faithfulness, completeness, and minimality. Our analysis shows that mean patching with performance-based evaluation most reliably isolates functionally correct circuits. In contrast, we demonstrate that direct logit attribution and probing classifiers primarily capture correlational features rather than causal ones, limiting their utility for circuit discovery. Overall, these results establish SR as a high-potential application domain for mechanistic interpretability and propose a principled methodology for circuit discovery.

翻译：随着Transformer在许多领域的成功应用，其在符号回归（SR）任务中也展现出显著效果；然而，其生成数学运算符的内部机制在很大程度上仍未得到探索。尽管机制可解释性方法已成功识别出语言和视觉模型中的功能电路，但尚未应用于符号回归领域。本文提出PATCHES——一种进化电路发现算法，能够为符号回归任务识别紧凑且正确的功能电路。通过PATCHES算法，我们分离出28个功能电路，首次实现了对符号回归Transformer的电路级表征。我们基于忠实性、完备性和最小性等关键概念构建了严谨的因果评估框架，验证了这些发现。分析表明，结合性能评估的均值修补方法能够最可靠地分离出功能正确的电路。相比之下，直接对数归因和探针分类器主要捕获的是相关性特征而非因果特征，这限制了它们在电路发现中的实用性。总体而言，本研究确立了符号回归作为机制可解释性研究的高潜力应用领域，并提出了一套系统化的电路发现方法论。