Automated program repair (APR) aims to help developers improve software reliability by generating patches for buggy programs. Although many code language models (CLM) are developed and effective in many software tasks such as code completion, there has been little comprehensive, in-depth work to evaluate CLMs' fixing capabilities and to fine-tune CLMs for the APR task. Firstly, this work is the first to evaluate ten CLMs on four APR benchmarks, which shows that surprisingly, the best CLM, as is, fixes 72% more bugs than the state-of-the-art deep-learning (DL)-based APR techniques. Secondly, one of the four APR benchmarks was created by us in this paper to avoid data leaking for a fair evaluation. Thirdly, it is the first work to fine-tune CLMs with APR training data, which shows that fine-tuning brings 31%-1,267% improvement to CLMs and enables them to fix 46%-164% more bugs than existing DL-based APR techniques. Fourthly, this work studies the impact of buggy lines, showing that CLMs, as is, cannot make good use of the buggy lines to fix bugs, yet fine-tuned CLMs could potentially over-rely on buggy lines. Lastly, this work analyzes the size, time, and memory efficiency of different CLMs. This work shows promising directions for the APR domain, such as fine-tuning CLMs with APR-specific designs, and also raises awareness of fair and comprehensive evaluations of CLMs and calls for more transparent reporting of open-source repositories used in the pre-training data to address the data leaking problem.

翻译：自动程序修复（APR）旨在通过为缺陷程序生成补丁来帮助开发者提升软件可靠性。尽管许多代码语言模型（CLM）在代码补全等软件任务中表现出色，但缺乏全面深入的评估工作来检验其修复能力，也鲜有研究针对APR任务对CLM进行微调。首先，本研究首次在四个APR基准上评估了十种CLM，结果出人意料地表明：未经调整的最佳CLM修复的缺陷数量比最先进的基于深度学习的APR技术多72%。其次，四个APR基准中有一个由本文创建，旨在避免数据泄露以实现公平评估。第三，本研究首次使用APR训练数据对CLM进行微调，结果发现微调可为CLM带来31%至1,267%的性能提升，使其修复的缺陷比现有基于深度学习的APR技术多46%至164%。第四，本研究分析了缺陷行的作用，发现未经微调的CLM难以有效利用缺陷行修复缺陷，而微调后的CLM可能过度依赖缺陷行。最后，本研究分析了不同CLM的规模、时间效率和内存效率。这项工作为APR领域指明了有前景的方向（例如针对APR特定设计微调CLM），同时呼吁对CLM进行公平全面的评估，并要求更透明地报告预训练数据中使用的开源仓库，以解决数据泄露问题。