Automated Program Repair (APR) techniques have shown more and more promising results in fixing real-world bugs. Despite the effectiveness, APR techniques still face an overfitting problem: a generated patch can be incorrect although it passes all tests. It is time-consuming to manually evaluate the correctness of generated patches that can pass all tests. To address this problem, many approaches have been proposed to automatically assess the correctness of patches generated by APR techniques. However, existing approaches require a large set of manually labeled patches as the training data. To mitigate the issue, in this study, we propose PatchZero, the patch correctness assessment by adopting large pre-trained models. Specifically, for patches generated by a new or unseen APR tool, PatchZero does not need labeled patches of this new or unseen APR tool for training (i.e., zero-shot) but directly queries the large pre-trained model to get predictions on the correctness labels without training. In this way, PatchZero can reduce the manual labeling effort when building a model to automatically assess the correctness of generated patches of new APR tools. To provide knowledge regarding the automatic patch correctness assessment (APCA) task to the large pre-trained models, we also design an instance-wise demonstration formation strategy by using contrastive learning. Specifically, PatchZero selects semantically similar patches to help the large pre-trained model to give more accurate predictions on the unlabeled patches. Our experimental results showed that PatchZero can achieve an accuracy of 82.7% and an F1-score of 86.0% on average although no labeled patch of the new or unseen APR tool is available. In addition, our proposed technique outperformed the prior state-of-the-art by a large margin.

翻译：自动程序修复（APR）技术在实际漏洞修复中展现出越来越有前景的效果。尽管有效，APR技术仍面临过拟合问题：生成的补丁虽然能通过所有测试，但可能不正确。人工评估这些通过全部测试的补丁的正确性非常耗时。为解决此问题，已提出多种方法自动评估APR技术生成的补丁正确性。然而，现有方法需要大量人工标注的补丁作为训练数据。为缓解这一局限，本研究提出PatchZero——一种利用大型预训练模型的补丁正确性评估方法。具体而言，对于新或未见过的APR工具生成的补丁，PatchZero无需该新工具的标注补丁进行训练（即零样本），而是直接查询大型预训练模型，无需训练即可获得正确性标签的预测。通过这种方式，PatchZero可在构建自动评估新APR工具生成补丁正确性的模型时减少人工标注工作。为向大型预训练模型提供自动补丁正确性评估（APCA）任务的相关知识，我们设计了一种基于对比学习的实例级演示构建策略。具体来说，PatchZero选择语义相似的补丁帮助大型预训练模型对未标注补丁给出更准确的预测。实验结果表明，尽管缺乏新或未见APR工具的标注补丁，PatchZero平均准确率可达82.7%，F1分数达86.0%。此外，本研究所提技术显著优于先前最先进方法。