Large language models (LLMs), such as OpenAI's Codex, have demonstrated their potential to generate code from natural language descriptions across a wide range of programming tasks. Several benchmarks have recently emerged to evaluate the ability of LLMs to generate functionally correct code from natural language intent with respect to a set of hidden test cases. This has enabled the research community to identify significant and reproducible advancements in LLM capabilities. However, there is currently a lack of benchmark datasets for assessing the ability of LLMs to generate functionally correct code edits based on natural language descriptions of intended changes. This paper aims to address this gap by motivating the problem NL2Fix of translating natural language descriptions of code changes (namely bug fixes described in Issue reports in repositories) into correct code fixes. To this end, we introduce Defects4J-NL2Fix, a dataset of 283 Java programs from the popular Defects4J dataset augmented with high-level descriptions of bug fixes, and empirically evaluate the performance of several state-of-the-art LLMs for the this task. Results show that these LLMS together are capable of generating plausible fixes for 64.6% of the bugs, and the best LLM-based technique can achieve up to 21.20% top-1 and 35.68% top-5 accuracy on this benchmark.

翻译：大型语言模型（LLMs），例如OpenAI的Codex，已在多种编程任务中展示了从自然语言描述生成代码的潜力。近期，多个基准测试被提出，用于评估LLMs根据自然语言意图生成功能正确代码的能力（针对一组隐藏测试用例）。这使研究界能够识别LLM能力的显著且可复现的进展。然而，目前仍缺乏用于评估LLMs基于自然语言变更描述生成功能正确代码编辑的基准数据集。本文旨在通过提出NL2Fix问题来填补这一空白，该问题将代码变更的自然语言描述（即代码仓库问题报告中描述的缺陷修复）转化为正确的代码修复。为此，我们引入Defects4J-NL2Fix数据集，该数据集包含来自流行Defects4J数据集的283个Java程序，并附有缺陷修复的高级描述，同时实证评估了多种最先进LLM在此任务上的性能。结果表明，这些LLM能够为64.6%的缺陷生成合理的修复方案，其中最佳基于LLM的技术在此基准测试上可达到21.20%的Top-1准确率和35.68%的Top-5准确率。