The rapid advancement in large language models (LLMs) has demonstrated significant potential in End-to-End Software Development (E2ESD). However, existing E2ESD benchmarks are limited by coarse-grained requirement specifications and unreliable evaluation protocols, hindering a true understanding of current framework capabilities. To address these limitations, we present E2EDev, a novel benchmark grounded in the principles of Behavior-Driven Development (BDD), which evaluates the capabilities of E2ESD frameworks by assessing whether the generated software meets user needs through mimicking real user interactions (Figure 1). E2EDev comprises (i) a fine-grained set of user requirements, (ii) multiple BDD test scenarios with corresponding Python step implementations for each requirement, and (iii) a fully automated testing pipeline built on the Behave framework. To ensure its quality while reducing the annotation effort, E2EDev leverages our proposed Human-in-the-Loop Multi-Agent Annotation Framework (HITL-MAA). By evaluating various E2ESD frameworks and LLM backbones with E2EDev, our analysis reveals a persistent struggle to effectively solve these tasks, underscoring the critical need for more effective and cost-efficient E2ESD solutions. Our codebase and benchmark are publicly available at https://github.com/SCUNLP/E2EDev.

翻译：大语言模型的快速发展在端到端软件开发（E2ESD）领域展现出显著潜力。然而，现有E2ESD基准测试受限于粗粒度的需求规范和不可靠的评估协议，阻碍了对当前框架能力的真实理解。为应对这些局限，我们提出E2EDev——一种基于行为驱动开发（BDD）原则的新型基准测试，通过模拟真实用户交互（图1）评估生成软件是否满足用户需求，从而检验E2ESD框架的能力。E2EDev包含：(i) 细粒度的用户需求集合；(ii) 针对每条需求的多个BDD测试场景及对应的Python步骤实现；(iii) 基于Behave框架构建的全自动化测试流水线。为在降低人工标注工作量的同时保证基准质量，E2EDev采用我们提出的人机协同多智能体标注框架（HITL-MAA）。通过使用E2EDev对多种E2ESD框架和大语言模型骨干架构进行评估，分析表明这些任务仍难以被有效解决，凸显了对更高效、更具成本效益的E2ESD解决方案的迫切需求。我们的代码库与基准数据已开源发布于 https://github.com/SCUNLP/E2EDev。