Recent advances in code agents have enabled automated software development at the project level, supported by large language models (LLMs). However, existing benchmarks for code agent evaluation face two major limitations. First, creating high-quality project-level evaluation datasets requires extensive domain expertise, leading to prohibitive annotation costs and limited diversity. Second, while recent Agent-as-a-Judge paradigms address the rigidity of traditional unit tests by enabling flexible metrics, their reliance on In-Context Learning (ICL) with general LLMs often results in inaccurate assessments that misalign with human standards. To address these challenges, we propose an agent-driven benchmark construction pipeline that leverages human supervision to efficiently generate diverse project-level tasks. Based on this, we introduce PRDBench, comprising 50 real-world Python projects across 20 domains, each with structured Product Requirement Documents (PRDs) and comprehensive criteria. Furthermore, to overcome the inaccuracy of general LLM judges, we propose a highly reliable evaluation framework powered by a specialized, fine-tuned model. Based on Qwen3-Coder-30B, our dedicated PRDJudge achieves over 90% human alignment in fixed-interface scenarios. Extensive experiments demonstrate that our suite provides a scalable, robust, and highly accurate framework for assessing state-of-the-art code agents.

翻译：近期代码代理的进展已使项目级自动化软件开发成为可能，这得益于大语言模型（LLMs）的支撑。然而，现有代码代理评估基准面临两大局限。首先，构建高质量项目级评估数据集需要广泛的领域专业知识，导致标注成本高昂且多样性受限。其次，尽管近期提出的"代理即裁判"范式通过引入灵活评估指标解决了传统单元测试僵化的问题，但其依赖通用大语言模型进行情境学习（ICL）的方式往往产生与人类标准偏差的不准确评估结果。针对这些挑战，我们提出一种代理驱动的基准构建流水线，通过人类监督高效生成多样化项目级任务。基于此，我们推出PRDBench基准，包含涵盖20个领域的50个真实世界Python项目，每个项目均配备结构化产品需求文档（PRD）与全面评估标准。此外，为解决通用大语言模型裁判不准确的问题，我们提出由专用微调模型驱动的高可靠评估框架。基于Qwen3-Coder-30B的专用裁判模型PRDJudge在固定接口场景下实现了超过90%的人类对齐率。大量实验表明，本套件为评估最先进代码代理提供了可扩展、稳健且高精度的框架。