How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs. To address the knowledge gap, we propose a new benchmark named DevEval, which has three advances. (1) DevEval aligns with real-world repositories in multiple dimensions, e.g., code distributions and dependency distributions. (2) DevEval is annotated by 13 developers and contains comprehensive annotations (e.g., requirements, original repositories, reference code, and reference dependencies). (3) DevEval comprises 1,874 testing samples from 117 repositories, covering 10 popular domains (e.g., Internet, Database). Based on DevEval, we propose repository-level code generation and evaluate 8 popular LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa). Our experiments reveal these LLMs' coding abilities in real-world code repositories. For example, in our experiments, the highest Pass@1 of gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize their shortcomings. We hope DevEval can facilitate the development of LLMs in real code repositories. DevEval, prompts, and LLMs' predictions have been released.

翻译：如何评估大型语言模型（LLM）的代码生成能力仍是一个开放问题。我们发现现有基准与真实世界代码仓库的对齐性较差，不足以充分评估LLM的代码能力。为弥补这一知识缺口，我们提出了名为DevEval的新基准，其具备三项优势。（1）DevEval在多个维度（如代码分布与依赖分布）上与真实世界仓库保持对齐。（2）DevEval由13名开发者共同标注，包含全面的标注信息（如需求描述、原始仓库、参考代码及参考依赖）。（3）DevEval包含来自117个仓库的1,874个测试样本，覆盖互联网、数据库等10个热门领域。基于DevEval，我们提出了仓库级代码生成任务，并在该基准上评估了8个主流LLM（如gpt-4、gpt-3.5、StarCoder 2、DeepSeek Coder、CodeLLaMa）。实验揭示了这些LLM在真实代码仓库中的编程能力表现。例如，在我们的实验中，gpt-4-turbo的最高Pass@1指标仅为53.04%。我们还分析了LLM的失败案例并归纳了其缺陷。我们希望DevEval能够促进LLM在真实代码仓库场景下的发展。DevEval基准、提示词及LLM生成结果均已开源发布。