The prominent large language models (LLMs) of today differ from past language models not only in size, but also in the fact that they are trained on a combination of natural language and formal language (code). As a medium between humans and computers, code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity. In this survey, we present an overview of the various benefits of integrating code into LLMs' training data. Specifically, beyond enhancing LLMs in code generation, we observe that these unique properties of code help (i) unlock the reasoning ability of LLMs, enabling their applications to a range of more complex natural language tasks; (ii) steer LLMs to produce structured and precise intermediate steps, which can then be connected to external execution ends through function calls; and (iii) take advantage of code compilation and execution environment, which also provides diverse feedback for model improvement. In addition, we trace how these profound capabilities of LLMs, brought by code, have led to their emergence as intelligent agents (IAs) in situations where the ability to understand instructions, decompose goals, plan and execute actions, and refine from feedback are crucial to their success on downstream tasks. Finally, we present several key challenges and future directions of empowering LLMs with code.

翻译：当今杰出的大语言模型（LLMs）与以往的语言模型不仅规模不同，还在于它们是基于自然语言与形式语言（代码）的混合语料进行训练的。作为人类与计算机之间的媒介，代码将高层目标转化为可执行的步骤，具备标准语法、逻辑一致性、抽象性和模块化特性。在本综述中，我们概述了将代码整合到LLM训练数据的诸多益处。具体而言，除了增强LLM的代码生成能力外，我们发现代码的这些独特属性有助于：（i）释放LLM的推理能力，使其能够应用于一系列更复杂的自然语言任务；（ii）引导LLM生成结构化且精确的中间步骤，进而通过函数调用连接到外部执行终端；（iii）利用代码编译和执行环境，为模型改进提供多样化的反馈。此外，我们追溯了代码赋予LLM的这些深刻能力如何使其在需要理解指令、分解目标、规划与执行行动以及从反馈中优化等关键环节的下游任务中，逐渐演变为智能代理（IA）。最后，我们提出了以代码赋能LLM面临的若干关键挑战与未来方向。