Planning is a crucial element of both human intelligence and contemporary large language models (LLMs). In this paper, we initiate a theoretical investigation into the emergence of planning capabilities in Transformer-based LLMs via their next-word prediction mechanisms. We model planning as a network path-finding task, where the objective is to generate a valid path from a specified source node to a designated target node. Our mathematical characterization shows that Transformer architectures can execute path-finding by embedding the adjacency and reachability matrices within their weights. Furthermore, our theoretical analysis of gradient-based learning dynamics reveals that LLMs can learn both the adjacency and a limited form of the reachability matrices. These theoretical insights are then validated through experiments, which demonstrate that Transformer architectures indeed learn the adjacency and an incomplete reachability matrices, consistent with our theoretical predictions. When applying our methodology to the real-world planning benchmark Blocksworld, our observations remain consistent. Additionally, our analyses uncover a fundamental limitation of current Transformer architectures in path-finding: these architectures cannot identify reachability relationships through transitivity, which leads to failures in generating paths when concatenation is required. These findings provide new insights into how the internal mechanisms of autoregressive learning facilitate intelligent planning and deepen our understanding of how future LLMs might achieve more advanced and general planning-and-reasoning capabilities across diverse applications.

翻译：规划是人类智能与当代大型语言模型（LLMs）的关键要素。本文通过理论探究，揭示了基于Transformer的LLMs如何通过其下一词预测机制涌现出规划能力。我们将规划建模为网络路径查找任务，其目标是在指定的源节点与目标节点之间生成有效路径。数学分析表明，Transformer架构能够通过在其权重中嵌入邻接矩阵与可达性矩阵来执行路径查找。进一步，基于梯度的学习动力学理论分析表明，LLMs能够同时学习邻接矩阵与一种受限形式的可达性矩阵。这些理论洞见通过实验得到验证：实验证明Transformer架构确实学习了邻接矩阵与不完整的可达性矩阵，与我们的理论预测一致。将我们的方法应用于现实世界规划基准测试Blocksworld时，观察结果保持一致性。此外，我们的分析揭示了当前Transformer架构在路径查找中的一个根本性局限：这些架构无法通过传递性识别可达关系，导致在需要路径拼接时生成路径失败。这些发现为理解自回归学习的内在机制如何促进智能规划提供了新视角，并深化了我们对未来LLMs如何在多样化应用中实现更先进、更通用的规划与推理能力的认知。