Large Language Models (LLMs) have shown promise in automated code generation but typically excel only in simpler tasks such as generating standalone code units. Real-world software development, however, often involves complex code repositories (named repo) with complex dependencies and extensive documentation. To fill this gap, our research pivots towards evaluating LLMs in a more realistic setting -- real-world repo-level code generation. We introduce CodeAgentBench, a manually curated benchmark for repo-level code generation. This benchmark comprises five high-quality Python projects, encompassing a total of 101 samples. We assess nine leading LLMs on repo-level tasks and observe a decline in their performance. To tackle this, we present CodeAgent, a novel LLM-based agent framework that employs external tools for effective repo-level code generation. CodeAgent integrates five programming tools, enabling interaction with software artifacts for information retrieval, code symbol navigation, and code testing. We implement four agent strategies to optimize these tools' usage. Our experiments on CodeAgentBench show that CodeAgent enhances LLM performance significantly, with improvements ranging from 18.1\% to 250\%. Further tests on the HumanEval benchmark confirm CodeAgent's adaptability and efficacy across various code generation tasks. Notably, CodeAgent outperforms commercial products like Github Copilot, showcasing superior accuracy and efficiency. These results demonstrate CodeAgent's robust capabilities in code generation, highlighting its potential for real-world repo-level coding challenges.

翻译：[摘要] 大语言模型（LLMs）在自动化代码生成方面展现出潜力，但通常仅擅长生成独立代码单元等简单任务。然而，真实软件开发常涉及具有复杂依赖关系和大量文档的复杂代码仓库（简称repo）。为填补这一空白，本研究转向在更真实的场景——真实仓库级代码生成中评估LLM性能。我们提出CodeAgentBench，一个手工构建的仓库级代码生成基准测试。该基准包含五个高质量Python项目，共涵盖101个样本。我们评估了九款领先LLM在仓库级任务中的表现，并观察到其性能下降。为解决这一问题，我们提出CodeAgent——一种基于LLM的新型代理框架，通过集成外部工具实现高效的仓库级代码生成。CodeAgent整合了五种编程工具，支持与软件制品交互以进行信息检索、代码符号导航和代码测试。我们设计了四种代理策略以优化这些工具的使用。在CodeAgentBench上的实验表明，CodeAgent显著提升了LLM性能，改进幅度达18.1%至250%。进一步在HumanEval基准上的测试证实了CodeAgent在各种代码生成任务中的适应性和有效性。值得注意的是，CodeAgent在准确性和效率上均优于GitHub Copilot等商业产品。这些结果展示了CodeAgent在代码生成中的强大能力，凸显了其应对真实仓库级编程挑战的潜力。