Large language models (LLMs) have demonstrated remarkable capabilities on a broad spectrum of downstream tasks. Within the realm of software engineering, specialized tasks on code, such as program repair, present unique challenges, necessitating fine-tuning to unlock state-of-the-art performance. Fine-tuning approaches proposed in the literature for LLMs on program repair tasks are however generally overlooking the need to reason about the logic behind code changes, beyond syntactic patterns in the data. High-performing fine-tuning experiments also usually come at very high computational costs. With MORepair, we propose a novel perspective on the learning focus of LLM fine-tuning for program repair: we not only adapt the LLM parameters to the syntactic nuances of the task of code transformation (objective 1), but we also specifically fine-tune the LLM with respect to the logical reason behind the code change in the training data (objective 2). Such a multi-objective fine-tuning will instruct LLMs to generate high-quality patches. We apply MORepair to fine-tune four open-source LLMs with different sizes and architectures. Experimental results on C++ and Java repair benchmarks show that the implemented fine-tuning effectively boosts LLM repair performance by 7.6% to 10% in Top-10 repair suggestions. We further show that our fine-tuning strategy yields superior performance compared to the incumbent state-of-the-art in fine-tuned models for program repair, Fine-tune-CoT and RepairLLaMA.

翻译：大语言模型（LLMs）已在广泛的下游任务中展现出卓越能力。在软件工程领域，程序修复等代码专项任务面临独特挑战，需要通过微调来发挥最佳性能。然而，现有文献中针对程序修复任务的LLM微调方法普遍忽视了对代码变更背后逻辑推理的需求，仅关注数据中的语法模式。同时，高性能的微调实验往往伴随高昂的计算成本。基于MORepair，我们提出了一种面向程序修复的LLM微调学习焦点新视角：不仅使LLM参数适配代码转换任务的语法特性（目标1），还专门针对训练数据中代码变更背后的逻辑推理进行微调（目标2）。这种多目标微调将指导LLM生成高质量补丁。我们对四种不同规模与架构的开源LLM应用MORepair进行微调。在C++和Java修复基准测试上的实验结果表明，该微调方法有效提升了LLM修复性能，在前10条修复建议中的准确率提升达7.6%至10%。进一步研究表明，我们的微调策略相比现有最先进的微调模型Fine-tune-CoT和RepairLLaMA展现出更优性能。