Humans write code in a fundamentally interactive manner and rely on constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recently exhibited promising coding capabilities, current coding benchmarks mostly consider a static instruction-to-code sequence transduction process, which has the potential for error propagation and a disconnect between the generated code and its final execution environment. To address this gap, we introduce InterCode, a lightweight, flexible, and easy-to-use framework of interactive coding as a standard reinforcement learning (RL) environment, with code as actions and execution feedback as observations. Our framework is language and platform agnostic, uses self-contained Docker environments to provide safe and reproducible execution, and is compatible out-of-the-box with traditional seq2seq coding methods, while enabling the development of new methods for interactive code generation. We use InterCode to create three interactive code environments with Bash, SQL, and Python as action spaces, leveraging data from the static NL2Bash, Spider, and MBPP datasets. We demonstrate InterCode's viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies such as ReAct and Plan & Solve. Our results showcase the benefits of interactive code generation and demonstrate that InterCode can serve as a challenging benchmark for advancing code understanding and generation capabilities. InterCode is designed to be easily extensible and can even be used to create new tasks such as Capture the Flag, a popular coding puzzle that is inherently multi-step and involves multiple programming languages. Project site with code and data: https://intercode-benchmark.github.io

翻译：人类编写代码本质上是交互式的，依赖持续的代码执行反馈来修正错误、消除歧义并分解任务。尽管大语言模型近期展现出令人瞩目的编码能力，但当前编码基准大多将编程视为静态的指令到代码序列转换过程，这可能导致错误累积，且生成的代码与其最终执行环境之间存在脱节。为解决这一问题，我们提出InterCode——一个轻量、灵活、易用的交互式编程框架，它被构建为标准强化学习环境，其中代码为动作，执行反馈为观测。该框架与语言和平台无关，通过自包含Docker环境提供安全且可复现的执行，既兼容传统序列到序列编码方法，又为交互式代码生成新方法的开发提供支持。我们利用InterCode创建了三个交互式代码环境（动作空间分别为Bash、SQL和Python），数据源自静态的NL2Bash、Spider和MBPP数据集。通过评估多个采用不同提示策略（如ReAct和Plan & Solve）的最先进大语言模型，我们验证了InterCode作为测试平台的可行性。实验结果表明交互式代码生成的优势，并证明InterCode可作为推动代码理解与生成能力发展的挑战性基准。InterCode设计上易于扩展，甚至可用于创建新任务，如夺旗赛——一种原生于多步骤、多语言场景的经典编程谜题。项目网站（含代码与数据）：https://intercode-benchmark.github.io