Software engineering agents have shown significant promise in writing code. As AI agents permeate code writing, and generate huge volumes of code automatically -- the matter of code quality comes front and centre. As the automatically generated code gets integrated into huge code-bases -- the issue of code review and broadly quality assurance becomes important. In this paper, we take a fresh look at the problem and curate a code review dataset for AI agents to work with. Our dataset called c-CRAB (pronounced see-crab) can evaluate agents for code review tasks. Specifically given a pull-request (which could be coming from code generation agents or humans), if a code review agent produces a review, our evaluation framework can asses the reviewing capability of the code review agents. Our evaluation framework is used to evaluate the state of the art today -- the open-source PR-agent, as well as commercial code review agents from Devin, Claude Code, and Codex. Our c-CRAB dataset is systematically constructed from human reviews -- given a human review of a pull request instance we generate corresponding tests to evaluate the code review agent generated reviews. Such a benchmark construction gives us several insights. Firstly, the existing review agents taken together can solve only around 40% of the c-CRAB tasks, indicating the potential to close this gap by future research. Secondly, we observe that the agent reviews often consider different aspects from the human reviews -- indicating the potential for human-agent collaboration for code review that could be deployed in future software teams. Last but not the least, the agent generated tests from our data-set act as a held out test-suite and hence quality gate for agent generated reviews. What this will mean for future collaboration of code generation agents, test generation agents and code review agents -- remains to be investigated.

翻译：软件工程智能体在代码编写方面已展现出显著潜力。当AI智能体渗透至代码编写领域并自动生成海量代码时，代码质量问题变得至关重要。随着自动生成的代码被集成到大型代码库中，代码审查及广义上的质量保证问题愈发重要。本文重新审视该问题，构建了一个供AI智能体使用的代码审查数据集。我们的数据集c-CRAB（音同"see-crab"）能够评估智能体在代码审查任务中的表现。具体而言，针对一个拉取请求（可能来自代码生成智能体或人类），若代码审查智能体生成审查意见，我们的评估框架可评估该智能体的审查能力。该评估框架被用于评估当前最新技术——包括开源PR-agent，以及来自Devin、Claude Code和Codex的商业代码审查智能体。c-CRAB数据集基于人类审查意见系统性构建：给定一个拉取请求实例的人类审查意见，我们生成相应的测试用例来评估代码审查智能体生成的审查质量。此类基准构建为我们提供了若干洞见。首先，现有审查智能体综合仅能解决约40%的c-CRAB任务，表明未来研究存在填补该差距的潜力。其次，我们观察到智能体审查通常关注与人类审查不同的维度，这暗示未来软件团队可部署人机协作进行代码审查。最后但同样重要的是，数据集生成的智能体测试可作为保留测试集，从而成为智能体生成审查意见的质量门禁。这对代码生成智能体、测试生成智能体与代码审查智能体未来协作模式的影响，仍有待深入探究。