This paper presents the results of finetuning large language models (LLMs) for the task of detecting vulnerabilities in source code. We leverage WizardCoder, a recent improvement of the state-of-the-art LLM StarCoder, and adapt it for vulnerability detection through further finetuning. To accelerate training, we modify WizardCoder's training procedure, also we investigate optimal training regimes. For the imbalanced dataset with many more negative examples than positive, we also explore different techniques to improve classification performance. The finetuned WizardCoder model achieves improvement in ROC AUC and F1 measures on balanced and imbalanced vulnerability datasets over CodeBERT-like model, demonstrating the effectiveness of adapting pretrained LLMs for vulnerability detection in source code. The key contributions are finetuning the state-of-the-art code LLM, WizardCoder, increasing its training speed without the performance harm, optimizing the training procedure and regimes, handling class imbalance, and improving performance on difficult vulnerability detection datasets. This demonstrates the potential for transfer learning by finetuning large pretrained language models for specialized source code analysis tasks.

翻译：本文介绍了针对源代码漏洞检测任务微调大型语言模型（LLM）的研究成果。我们利用WizardCoder（近期基于前沿LLM StarCoder改进的模型），通过进一步微调使其适配漏洞检测任务。为加速训练，我们改进了WizardCoder的训练流程，并探究了最优训练方案。针对负例远多于正例的非平衡数据集，我们还探索了多种提升分类性能的技术。微调后的WizardCoder模型在平衡与非平衡漏洞数据集上的ROC AUC和F1指标均优于CodeBERT类模型，证明了将预训练LLM适配于源代码漏洞检测的有效性。核心贡献包括：微调前沿代码LLM WizardCoder、在不损失性能的前提下提升其训练速度、优化训练流程与方案、处理类别不平衡问题，以及在困难漏洞检测数据集上提升性能。这展示了通过微调大型预训练语言模型实现迁移学习，以专门应对源代码分析任务的潜力。