Humans write code in a fundamentally interactive manner and rely on constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recently exhibited promising coding capabilities, current coding benchmarks mostly consider a static instruction-to-code sequence transduction process, which has the potential for error propagation and a disconnect between the generated code and its final execution environment. To address this gap, we introduce InterCode, a lightweight, flexible, and easy-to-use framework of interactive coding as a standard reinforcement learning (RL) environment, with code as actions and execution feedback as observations. Our framework is language and platform agnostic, uses self-contained Docker environments to provide safe and reproducible execution, and is compatible out-of-the-box with traditional seq2seq coding methods, while enabling the development of new methods for interactive code generation. We use InterCode to create two interactive code environments with Bash and SQL as action spaces, leveraging data from the static Spider and NL2Bash datasets. We demonstrate InterCode's viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies such as ReAct and Plan & Solve. Our results showcase the benefits of interactive code generation and demonstrate that InterCode can serve as a challenging benchmark for advancing code understanding and generation capabilities. InterCode is designed to be easily extensible and can even be used to incorporate new tasks such as Capture the Flag, a popular coding puzzle that is inherently multi-step and involves multiple programming languages. Project site with code and data: https://intercode-benchmark.github.io

翻译：摘要：人类编写代码本质上是一种交互式过程，依赖持续的执性反馈来纠正错误、消除歧义并分解任务。尽管大语言模型近期展现出卓越的编码能力，但当前的编程基准大多将任务建模为从静态指令到代码的序列转换过程，这种方法存在错误传播风险，且生成的代码与最终执行环境之间存在脱节。为解决这一问题，我们提出InterCode——一个轻量级、灵活且易用的交互式编程框架，将其构建为标准强化学习环境，其中代码作为动作，执行反馈作为观测。该框架与语言及平台无关，采用自包含的Docker环境提供安全且可复现的执行过程，既可直接兼容传统序列到序列编码方法，又能支持开发新型交互式代码生成方法。我们利用InterCode创建了两个交互式代码环境，以Bash和SQL作为动作空间，并基于静态Spider与NL2Bash数据集进行构建。通过评估多个采用不同提示策略（如ReAct与Plan & Solve）的最先进大语言模型，我们验证了InterCode作为测试平台的可行性。实验结果表明交互式代码生成的优势，并证明InterCode可作为推动代码理解与生成能力发展的挑战性基准。InterCode设计上易于扩展，甚至可融入如夺旗赛（Capture the Flag）等新型任务——这类流行编程谜题天然具有多步骤特性，且涉及多种编程语言。项目网站（含代码与数据）：https://intercode-benchmark.github.io