Algorithm extraction aims to synthesize executable programs directly from models trained on specific algorithmic tasks, enabling de novo algorithm discovery without relying on human-written code. However, extending this paradigm to Transformer is hindered by superposition, where entangled features encoded in overlapping directions obstruct the extraction of symbolic expressions. In this work, we propose the Discrete Transformer, an architecture explicitly engineered to bridge the gap between continuous representations and discrete symbolic logic. By enforcing a strict functional disentanglement, which constrains Numerical Attention to information routing and Numerical MLP to element-wise arithmetic, and employing temperature-annealed sampling, our method effectively facilitates the extraction of human-readable programs. Empirically, the Discrete Transformer not only achieves performance comparable to RNN-based baselines but crucially extends interpretability to continuous variable domains. Moreover, our analysis of the annealing process shows that the efficient discrete search undergoes a clear phase transition from exploration to exploitation. We further demonstrate that our method enables fine-grained control over synthesized programs by imposing inductive biases. Collectively, these findings establish the Discrete Transformer as a robust framework for demonstration-free algorithm discovery, offering a rigorous pathway toward Transformer interpretability.

翻译：算法提取旨在直接从针对特定算法任务训练的模型中合成可执行程序，从而无需依赖人工编写的代码即可实现从零开始的算法发现。然而，将这一范式扩展到Transformer模型受到叠加现象的阻碍，其中编码在重叠方向上的纠缠特征阻碍了符号表达式的提取。在本工作中，我们提出了离散Transformer，这是一种专门设计的架构，旨在弥合连续表示与离散符号逻辑之间的鸿沟。通过强制实施严格的功能解耦——将数值注意力机制约束于信息路由，将数值多层感知机约束于逐元素算术运算——并采用温度退火采样，我们的方法有效促进了人类可读程序的提取。实证结果表明，离散Transformer不仅实现了与基于RNN的基线模型相当的性能，而且关键地将可解释性扩展到了连续变量领域。此外，我们对退火过程的分析表明，高效的离散搜索经历了从探索到利用的清晰相变。我们进一步证明，通过施加归纳偏置，我们的方法能够对合成程序进行细粒度控制。综上所述，这些发现确立了离散Transformer作为一个无需演示的算法发现的稳健框架，为Transformer的可解释性研究提供了一条严谨的路径。