The prominent large language models (LLMs) of today differ from past language models not only in size, but also in the fact that they are trained on a combination of natural language and formal language (code). As a medium between humans and computers, code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity. In this survey, we present an overview of the various benefits of integrating code into LLMs' training data. Specifically, beyond enhancing LLMs in code generation, we observe that these unique properties of code help (i) unlock the reasoning ability of LLMs, enabling their applications to a range of more complex natural language tasks; (ii) steer LLMs to produce structured and precise intermediate steps, which can then be connected to external execution ends through function calls; and (iii) take advantage of code compilation and execution environment, which also provides diverse feedback for model improvement. In addition, we trace how these profound capabilities of LLMs, brought by code, have led to their emergence as intelligent agents (IAs) in situations where the ability to understand instructions, decompose goals, plan and execute actions, and refine from feedback are crucial to their success on downstream tasks. Finally, we present several key challenges and future directions of empowering LLMs with code.

翻译：当今卓越的大语言模型（LLM）与过去的语言模型相比，不仅参数规模更大，更关键在于它们的训练数据融合了自然语言与形式语言（代码）。作为人类与计算机之间的媒介，代码将高层目标转化为可执行的步骤，具备标准化语法、逻辑一致性、抽象性和模块化特性。在本综述中，我们系统梳理了将代码融入LLM训练数据的多重优势。具体而言，除了提升LLM的代码生成能力外，我们观察到代码的这些独特属性有助于：（i）解锁LLM的推理能力，使其能应用于更复杂的自然语言任务；（ii）引导LLM生成结构化且精准的中间步骤，进而通过函数调用连接外部执行终端；（iii）利用代码编译与执行环境，为模型优化提供多样化的反馈机制。此外，我们追溯了代码赋予LLM的这些深层能力如何催生其成为智能代理（IAs）——在理解指令、分解目标、规划执行动作以及从反馈中优化的能力对下游任务成功至关重要的场景中。最后，我们提出了用代码赋能LLM的关键挑战与未来方向。