Recent advancements in large language models (LLMs) have significantly enhanced their coding capabilities. However, existing benchmarks predominantly focused on simplified or isolated aspects of programming, such as single-file code generation or repository issue debugging, falling short of measuring the full spectrum of challenges raised by real-world programming activities. To this end, we propose DevBench, a comprehensive benchmark that evaluates LLMs across various stages of the software development lifecycle, including software design, environment setup, implementation, acceptance testing, and unit testing. DevBench features a wide range of programming languages and domains, high-quality data collection, and carefully designed and verified metrics for each task. Empirical studies show that current LLMs, including GPT-4-Turbo, fail to solve the challenges presented within DevBench. Analyses reveal that models struggle with understanding the complex structures in the repository, managing the compilation process, and grasping advanced programming concepts. Our findings offer actionable insights for the future development of LLMs toward real-world programming applications. Our benchmark is available at https://github.com/open-compass/DevBench

翻译：近年来，大型语言模型（LLMs）的编码能力得到显著提升。然而，现有基准测试主要关注编程的简化或孤立方面，例如单文件代码生成或仓库问题调试，未能全面衡量真实编程活动带来的各类挑战。为此，我们提出DevBench——一个涵盖软件开发生命周期各阶段的综合基准测试，包括软件设计、环境搭建、实现、验收测试和单元测试。DevBench涵盖多种编程语言和领域，采用高质量数据收集方式，并为每项任务设计了经过精心验证的评估指标。实证研究表明，包括GPT-4-Turbo在内的当前LLMs均无法解决DevBench所呈现的挑战。分析揭示，模型在理解仓库中复杂结构、管理编译过程以及掌握高级编程概念方面存在困难。我们的发现为LLMs面向真实编程应用的未来发展提供了可行见解。基准测试已开源至https://github.com/open-compass/DevBench