Code generation models based on the pre-training and fine-tuning paradigm have been increasingly attempted by both academia and industry, resulting in well-known industrial models such as Codex, CodeGen, and PanGu-Coder. To validate the performance of these models, multiple existing benchmarks (e.g., AiXBench and HumanEval) are proposed, including only cases of generating a standalone function, i.e., a function that invokes or accesses only built-in functions and standard libraries. However, standalone functions constitute only about 30\% of functions from real open-source projects. To assess a model's performance for pragmatic code generation (i.e., code generation for real settings of open source or proprietary code), in this paper, we propose a benchmark named CoderEval of pragmatic code generation with generative pre-trained models. Compared with the widely-used HumanEval benchmark from OpenAI, CoderEval can be used to assess the performance of models against pragmatic code generation beyond just generating standalone functions. Through the evaluation of three public available models (CodeGen, PanGu-Coder, and Codex) on CoderEval, we analyze and discuss the current progress and future directions of pragmatic code generation with a generative pre-trained model.

翻译：基于预训练与微调范式的代码生成模型日益受到学术界与工业界的探索，催生了如Codex、CodeGen和PanGu-Coder等知名工业模型。为验证这些模型的性能，现有基准（如AiXBench和HumanEval）仅包含生成独立函数的案例，即仅调用或访问内置函数和标准库的函数。然而，独立函数仅占真实开源项目中函数的约30%。为评估模型在实用代码生成（即面向开源或专有代码真实场景的代码生成）中的表现，本文提出名为CoderEval的基准，用于基于生成式预训练模型的实用代码生成。与OpenAI广泛使用的HumanEval基准相比，CoderEval可评估模型在超越独立函数生成之外的实用代码生成性能。通过使用CoderEval对三种公开模型（CodeGen、PanGu-Coder和Codex）进行评估，我们分析并讨论了当前基于生成式预训练模型的实用代码生成进展及未来方向。