Code review is an effective software quality assurance activity; however, it is labor-intensive and time-consuming. Thus, a number of generation-based automatic code review (ACR) approaches have been proposed recently, which leverage deep learning techniques to automate various activities in the code review process (e.g., code revision generation and review comment generation). We find the previous works carry three main limitations. First, the ACR approaches have been shown to be beneficial in each work, but those methods are not comprehensively compared with each other to show their superiority over their peer ACR approaches. Second, general-purpose pre-trained models such as CodeT5 are proven to be effective in a wide range of Software Engineering (SE) tasks. However, no prior work has investigated the effectiveness of these models in ACR tasks yet. Third, prior works heavily rely on the Exact Match (EM) metric which only focuses on the perfect predictions and ignores the positive progress made by incomplete answers. To fill such a research gap, we conduct a comprehensive study by comparing the effectiveness of recent ACR tools as well as the general-purpose pre-trained models. The results show that a general-purpose pre-trained model CodeT5 can outperform other models in most cases. Specifically, CodeT5 outperforms the prior state-of-the-art by 13.4\%--38.9\% in two code revision generation tasks. In addition, we introduce a new metric namely Edit Progress (EP) to quantify the partial progress made by ACR tools. The results show that the rankings of models for each task could be changed according to whether EM or EP is being utilized. Lastly, we derive several insightful lessons from the experimental results and reveal future research directions for generation-based code review automation.

翻译：代码审查是一种有效的软件质量保证活动；然而，它劳动密集且耗时。因此，最近提出了许多基于生成的自动代码审查（ACR）方法，这些方法利用深度学习技术来自动化代码审查过程中的各种活动（例如，代码修订生成和审查评论生成）。我们发现先前的工作存在三个主要局限性。首先，ACR方法在每项工作中已被证明是有益的，但这些方法并未进行全面相互比较，以显示其相对于同类ACR方法的优越性。其次，通用预训练模型（如CodeT5）已被证明在广泛的软件工程（SE）任务中有效。然而，先前没有工作研究这些模型在ACR任务中的有效性。第三，先前的工作严重依赖精确匹配（EM）指标，该指标仅关注完美预测，而忽略了不完整答案所取得的积极进展。为填补这一研究空白，我们通过比较近期ACR工具以及通用预训练模型的有效性，进行了一项全面研究。结果表明，通用预训练模型CodeT5在大多数情况下可以超越其他模型。具体而言，在两个代码修订生成任务中，CodeT5的性能比先前的领先水平高出13.4%–38.9%。此外，我们引入了一个新指标，即编辑进度（EP），用于量化ACR工具取得的局部进展。结果表明，根据使用的是EM还是EP，每个任务的模型排名可能会发生变化。最后，我们从实验结果中得出了一些有价值的教训，并揭示了基于生成的代码审查自动化的未来研究方向。