The execution of graph algorithms using neural networks has recently attracted significant interest due to promising empirical progress. This motivates further understanding of how neural networks can replicate reasoning steps with relational data. In this work, we study the ability of transformer networks to simulate algorithms on graphs from a theoretical perspective. The architecture that we utilize is a looped transformer with extra attention heads that interact with the graph. We prove by construction that this architecture can simulate algorithms such as Dijkstra's shortest path algorithm, Breadth- and Depth-First Search, and Kosaraju's strongly connected components algorithm. The width of the network does not increase with the size of the input graph, which implies that the network can simulate the above algorithms for any graph. Despite this property, we show that there is a limit to simulation in our solution due to finite precision. Finally, we show a Turing Completeness result with constant width when the extra attention heads are utilized.

翻译：利用神经网络执行图算法近期因实验上的显著进展而引起了广泛兴趣。这进一步激发了人们对神经网络如何通过关系数据复现推理步骤的理解。本文从理论角度研究Transformer网络模拟图算法的能力。我们使用的架构是一种带有与图交互的额外注意力头的循环Transformer。通过构造证明，该架构能够模拟迪杰斯特拉最短路径算法、广度优先搜索、深度优先搜索以及科萨拉朱强连通分量算法等算法。网络宽度不会随输入图的规模增加，这意味着该网络可模拟任意图的上述算法。尽管具备这一特性，我们证明由于有限精度，该解决方案中的模拟能力存在限制。最后，我们展示了在利用额外注意力头时，常数宽度下的图灵完备性结果。