In this paper, we present the findings of our Project ALPINE which stands for ``Autoregressive Learning for Planning In NEtworks." Project ALPINE initiates a theoretical investigation into the development of planning capabilities in Transformer-based language models through their autoregressive learning mechanisms, aiming to identify any potential limitations in their planning abilities. We abstract planning as a network path-finding task where the objective is to generate a valid path from a specified source node to a designated target node. In terms of expressiveness, we show that the Transformer is capable of executing path-finding by embedding the adjacency and reachability matrices within its weights. Our theoretical analysis of the gradient-based learning dynamic of the Transformer reveals that the Transformer is capable of learning both the adjacency matrix and a limited form of the reachability matrix. These theoretical insights are then validated through experiments, which demonstrate that the Transformer indeed learns the adjacency matrix and an incomplete reachability matrix, which aligns with the predictions made in our theoretical analysis. Additionally, when applying our methodology to a real-world planning benchmark, called Blocksworld, our observations remain consistent. Our theoretical and empirical analyses further unveil a potential limitation of Transformer in path-finding: it cannot identify reachability relationships through transitivity, and thus would fail when path concatenation is needed to generate a path. In summary, our findings shed new light on how the internal mechanisms of autoregressive learning enable planning in networks. This study may contribute to our understanding of the general planning capabilities in other related domains.

翻译：在本文中，我们介绍了ALPINE项目（全称为“Autoregressive Learning for Planning In NEtworks”，即“网络规划中的自回归学习”）的研究成果。ALPINE项目从理论上探讨了基于Transformer的语言模型通过自回归学习机制发展规划能力的机理，旨在识别其规划能力可能存在的局限性。我们将规划任务抽象为网络路径查找问题，其目标是从指定源节点生成一条通向目标节点的有效路径。在表达能力方面，我们证明Transformer能够通过将邻接矩阵和可达性矩阵嵌入其权重中来执行路径查找。对Transformer基于梯度的学习动力学的理论分析表明，Transformer能够同时学习邻接矩阵和有限形式的可达性矩阵。这些理论见解随后通过实验得到验证，实验证明Transformer确实学习了邻接矩阵和不完整的可达性矩阵，这与我们理论分析中的预测一致。此外，当我们将该方法应用于名为Blocksworld的现实世界规划基准时，观察结果仍保持一致。我们的理论和实证分析进一步揭示了Transformer在路径查找中的一个潜在局限性：它无法通过传递性识别可达关系，因此在需要路径拼接生成路径时会失败。总之，我们的发现为自回归学习的内部机制如何实现网络规划提供了新的见解。这项研究可能有助于理解其他相关领域中通用规划能力的基础。