Humans write code in a fundamentally interactive manner and rely on constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recently exhibited promising coding capabilities, current coding benchmarks mostly consider a static instruction-to-code sequence transduction process, which has the potential for error propagation and a disconnect between the generated code and its final execution environment. To address this gap, we introduce InterCode, a lightweight, flexible, and easy-to-use framework of interactive coding as a standard reinforcement learning (RL) environment, with code as actions and execution feedback as observations. Our framework is language and platform agnostic, uses self-contained Docker environments to provide safe and reproducible execution, and is compatible out-of-the-box with traditional seq2seq coding methods, while enabling the development of new methods for interactive code generation. We use InterCode to create two interactive code environments with Bash and SQL as action spaces, leveraging data from the static Spider and NL2Bash datasets. We demonstrate InterCode's viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies such as ReAct and Plan & Solve. Our results showcase the benefits of interactive code generation and demonstrate that InterCode can serve as a challenging benchmark for advancing code understanding and generation capabilities. InterCode is designed to be easily extensible and can even be used to incorporate new tasks such as Capture the Flag, a popular coding puzzle that is inherently multi-step and involves multiple programming languages. Project site with code and data: https://intercode-benchmark.github.io

翻译：人类编写代码本质上是一个交互过程，依赖持续的执行反馈来纠正错误、消除歧义和分解任务。尽管大语言模型近期展现了出色的编码能力，但当前的编码基准测试大多将任务建模为静态的指令到代码序列转换过程，这可能导致错误传播，且生成的代码与其最终执行环境之间存在脱节。为解决这一问题，我们提出InterCode——一个轻量级、灵活且易用的交互式编码框架，将其标准化为强化学习环境，其中代码作为动作、执行反馈作为观测。该框架与语言和平台无关，通过自包含的Docker环境提供安全且可复现的执行，天然兼容传统的序列到序列编码方法，同时支持开发交互式代码生成的新方法。我们利用InterCode基于静态Spider和NL2Bash数据集构建了两个交互式代码环境，以Bash和SQL作为动作空间。通过评估采用不同提示策略（如ReAct与Plan & Solve）的多个先进大语言模型，我们证明了InterCode作为测试平台的可行性。结果表明交互式代码生成具有显著优势，且InterCode可作为推动代码理解与生成能力发展的挑战性基准。该框架设计易于扩展，甚至可纳入新任务（如“夺旗”这类本质上需要多步骤且涉及多种编程语言的流行编码谜题）。项目网站及代码与数据参见：https://intercode-benchmark.github.io