Modern neural network architectures still struggle to learn algorithmic procedures that require to systematically apply compositional rules to solve out-of-distribution problem instances. In this work, we focus on formula simplification problems, a class of synthetic benchmarks used to study the systematic generalization capabilities of neural architectures. We propose a modular architecture designed to learn a general procedure for solving nested mathematical formulas by only relying on a minimal set of training examples. Inspired by rewriting systems, a classic framework in symbolic artificial intelligence, we include in the architecture three specialized and interacting modules: the Selector, trained to identify solvable sub-expressions; the Solver, mapping sub-expressions to their values; and the Combiner, replacing sub-expressions in the original formula with the solution provided by the Solver. We benchmark our system against the Neural Data Router, a recent model specialized for systematic generalization, and a state-of-the-art large language model (GPT-4) probed with advanced prompting strategies. We demonstrate that our approach achieves a higher degree of out-of-distribution generalization compared to these alternative approaches on three different types of formula simplification problems, and we discuss its limitations by analyzing its failures.

翻译：现代神经网络架构在学习需要系统性地应用组合规则以解决分布外问题实例的算法程序方面仍然面临困难。在本研究中，我们聚焦于公式简化问题——一类用于研究神经架构系统性泛化能力的合成基准。我们提出了一种模块化架构，旨在仅依赖少量训练示例来学习解决嵌套数学公式的通用过程。受符号人工智能经典框架——重写系统的启发，我们在架构中设计了三个专门化且相互作用的模块：经过训练以识别可解子表达式的选择器（Selector）；将子表达式映射到其值的求解器（Solver）；以及将原始公式中的子表达式替换为求解器所提供解的合成器（Combiner）。我们将本系统与专门用于系统性泛化的最新模型——神经数据路由器（Neural Data Router），以及采用高级提示策略进行探测的先进大语言模型（GPT-4）进行了基准测试。实验表明，在三种不同类型的公式简化问题上，我们的方法相比这些替代方案实现了更高程度的分布外泛化，并通过分析其失败案例探讨了该方法的局限性。