In this paper, we present the findings of our Project ALPINE which stands for ``Autoregressive Learning for Planning In NEtworks." Project ALPINE initiates a theoretical investigation into the development of planning capabilities in Transformer-based language models through their autoregressive learning mechanisms, aiming to identify any potential limitations in their planning abilities. We abstract planning as a network path-finding task where the objective is to generate a valid path from a specified source node to a designated target node. In terms of expressiveness, we show that the Transformer is capable of executing path-finding by embedding the adjacency and reachability matrices within its weights. Our theoretical analysis of the gradient-based learning dynamic of the Transformer reveals that the Transformer is capable of learning both the adjacency matrix and a limited form of the reachability matrix. These theoretical insights are then validated through experiments, which demonstrate that the Transformer indeed learns the adjacency matrix and an incomplete reachability matrix, which aligns with the predictions made in our theoretical analysis. Additionally, when applying our methodology to a real-world planning benchmark, called Blocksworld, our observations remain consistent. Our theoretical and empirical analyses further unveil a potential limitation of Transformer in path-finding: it cannot identify reachability relationships through transitivity, and thus would fail when path concatenation is needed to generate a path. In summary, our findings shed new light on how the internal mechanisms of autoregressive learning enable planning in networks. This study may contribute to our understanding of the general planning capabilities in other related domains.

翻译：本文介绍了我们的研究项目ALPINE（Autoregressive Learning for Planning In NEtworks，即“网络中规划的自回归学习”）的发现。该项目从理论层面探究了基于Transformer的语言模型如何通过其自回归学习机制发展规划能力，旨在揭示其规划能力可能存在的局限性。我们将规划抽象为网络路径寻找任务，其目标是在指定源节点与目标节点之间生成有效路径。在表达能力方面，我们证明了Transformer能够通过在其权重中嵌入邻接矩阵与可达性矩阵来执行路径寻找。通过对Transformer基于梯度的学习动态进行理论分析，我们发现Transformer能够学习邻接矩阵以及一种受限形式的可达性矩阵。这些理论见解随后通过实验得到验证：实验表明Transformer确实学习了邻接矩阵以及一个不完整的可达性矩阵，这与我们的理论预测一致。此外，当我们将该方法应用于名为Blocksworld的真实世界规划基准测试时，观察结果依然保持一致。我们的理论与实证分析进一步揭示了Transformer在路径寻找中的一个潜在局限：它无法通过传递性识别可达关系，因此在需要路径拼接以生成路径时会失败。总之，我们的研究结果为自回归学习的内部机制如何实现网络中的规划提供了新的见解。这项研究可能有助于我们理解其他相关领域中的通用规划能力。