Code generation models based on the pre-training and fine-tuning paradigm have been increasingly attempted by both academia and industry, resulting in well-known industrial models such as Codex, CodeGen, and PanGu-Coder. To evaluate the effectiveness of these models, multiple existing benchmarks are proposed, including only cases of generating a standalone function, i.e., a function that may invoke or access only built-in functions and standard libraries. However, non-standalone functions, which typically are not included in the existing benchmarks, constitute more than 70% of the functions in popular open-source projects, and evaluating models' effectiveness on standalone functions cannot reflect these models' effectiveness on pragmatic code generation scenarios. To help bridge the preceding gap, in this paper, we propose a benchmark named CoderEval, consisting of 230 Python and 230 Java code generation tasks carefully curated from popular real-world open-source projects and a self-contained execution platform to automatically assess the functional correctness of generated code. CoderEval supports code generation tasks from six levels of context dependency, where context refers to code elements such as types, APIs, variables, and consts defined outside the function under generation but within the dependent third-party libraries, current class, file, or project. CoderEval can be used to evaluate the effectiveness of models in generating code beyond only standalone functions. By evaluating three code generation models on CoderEval, we find that the effectiveness of these models in generating standalone functions is substantially higher than that in generating non-standalone functions. Our analysis highlights the current progress and pinpoints future directions to further improve a model's effectiveness by leveraging contextual information for pragmatic code generation.

翻译：基于预训练与微调范式的代码生成模型正日益受到学术界和工业界的探索，催生了诸如Codex、CodeGen和PanGu-Coder等知名工业模型。为评估这些模型的有效性，现有基准仅包含独立函数的生成案例（即仅调用或访问内置函数与标准库的函数）。然而，非独立函数在主流开源项目中占比超过70%，却通常未被纳入现有基准。仅通过独立函数评估模型的有效性，无法反映其在实用代码生成场景中的表现。为弥合上述差距，本文提出名为CoderEval的基准，包含从真实世界流行开源项目中精心筛选的230个Python和230个Java代码生成任务，并配备自包含的执行平台以自动评估生成代码的功能正确性。CoderEval支持六种上下文依赖级别的代码生成任务——上下文指代待生成函数外部但在依赖的第三方库、当前类、文件或项目内部定义的代码元素（如类型、API、变量和常量）。该基准可用于评估模型生成独立函数以外代码的有效性。通过对三个代码生成模型在CoderEval上的评估，我们发现这些模型在生成独立函数时的有效性显著高于生成非独立函数。我们的分析既揭示了当前进展，也为通过利用上下文信息提升实用代码生成模型的有效性指明了未来方向。