Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+, while none of them could on HumanEval. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.

翻译：程序合成长期以来一直是研究热点，近期方法主要关注直接利用大语言模型（LLMs）的能力来生成代码。编程基准测试通过设计好的合成问题与测试用例，衡量不同LLMs在代码合成中的性能。然而，现有测试用例在数量和质量上均存在局限性，难以充分评估生成代码的功能正确性。这种局限性引出一个核心问题：在LLM时代，生成的代码真的正确吗？为解答此问题，我们提出EvalPlus——一种用于严格基准测试LLM合成代码功能正确性的代码合成评估框架。EvalPlus通过自动化测试输入生成器（同时基于LLM和变异策略）生成大量新测试用例，对现有评估数据集进行增强。作为通用框架，EvalPlus将广受欢迎的HumanEval基准测试的用例数扩展80倍，构建出HumanEval+。我们对26种主流LLMs（如GPT-4和ChatGPT）的广泛评估表明，HumanEval+能够捕获大量先前未被检测到的LLM错误生成代码，使pass@k指标降低高达19.3%-28.9%。令人意外的是，我们还发现测试用例不足可能导致排名偏差：例如，WizardCoder-CodeLlama和Phind-CodeLlama在HumanEval+上均超越ChatGPT，而二者在原始HumanEval上均未实现此结果。本研究不仅揭示此前流行的代码合成评估结果未能准确反映LLMs的真实性能，更开辟了通过自动化测试改进编程基准的新方向。我们已开源相关工具、增强数据集及所有LLM生成代码（访问https://github.com/evalplus/evalplus），以促进和加速未来LLM代码生成研究。