We investigate the extent to which Large Language Models (LLMs) can simulate the execution of computer code and algorithms. We begin by looking at straight line programs, and show that current LLMs demonstrate poor performance even with such simple programs -- performance rapidly degrades with the length of code. We then investigate the ability of LLMs to simulate programs that contain critical paths and redundant instructions. We also go beyond straight line program simulation with sorting algorithms and nested loops, and we show the computational complexity of a routine directly affects the ability of an LLM to simulate its execution. We observe that LLMs execute instructions sequentially and with a low error margin only for short programs or standard procedures. LLMs' code simulation is in tension with their pattern recognition and memorisation capabilities: on tasks where memorisation is detrimental, we propose a novel prompting method to simulate code execution line by line. Empirically, our new Chain of Simulation (CoSm) method improves on the standard Chain of Thought prompting approach by avoiding the pitfalls of memorisation.

翻译：我们研究了大型语言模型（LLMs）模拟计算机代码和算法执行的程度。首先从直线型程序入手，发现当前LLMs即使处理这类简单程序也表现不佳——其性能随代码长度增加而迅速下降。接着探讨了LLMs模拟含有关键路径和冗余指令的程序的能力，并进一步研究了排序算法和嵌套循环等超越直线型程序的模拟场景。实验表明，程序的计算复杂度直接影响LLMs模拟执行的能力。我们观察到LLMs仅对短程序或标准流程能实现低错误率的顺序指令执行。LLMs的代码模拟能力与其模式识别和记忆能力存在矛盾：在记忆会干扰任务时，我们提出了一种逐行模拟代码执行的新型提示方法。实验证明，我们的新方法"模拟链"（Chain of Simulation, CoSm）通过规避记忆缺陷，在标准思维链提示方法基础上实现了性能提升。