This paper presents the results of finetuning large language models (LLMs) for the task of detecting vulnerabilities in source code. We leverage WizardCoder, a recent improvement of the state-of-the-art LLM StarCoder, and adapt it for vulnerability detection through further finetuning. To accelerate training, we modify WizardCoder's training procedure, also we investigate optimal training regimes. For the imbalanced dataset with many more negative examples than positive, we also explore different techniques to improve classification performance. The finetuned WizardCoder model achieves improvement in ROC AUC and F1 measures on balanced and imbalanced vulnerability datasets over CodeBERT-like model, demonstrating the effectiveness of adapting pretrained LLMs for vulnerability detection in source code. The key contributions are finetuning the state-of-the-art code LLM, WizardCoder, increasing its training speed without the performance harm, optimizing the training procedure and regimes, handling class imbalance, and improving performance on difficult vulnerability detection datasets. This demonstrates the potential for transfer learning by finetuning large pretrained language models for specialized source code analysis tasks.

翻译：本文介绍了微调大型语言模型（LLMs）以执行源代码漏洞检测任务的结果。我们利用了WizardCoder——最新改进的先进LLM StarCoder，并通过进一步微调将其适配于漏洞检测。为加速训练，我们修改了WizardCoder的训练流程，并研究了最优训练方案。针对负样本远多于正样本的不平衡数据集，我们还探索了多种提升分类性能的技术。微调后的WizardCoder模型在平衡与不平衡漏洞数据集上的ROC AUC和F1指标上均优于CodeBERT类模型，展现了将预训练LLMs适配于源代码漏洞检测的有效性。主要贡献包括：微调先进代码LLM WizardCoder、在不损害性能的情况下提升其训练速度、优化训练流程与方案、处理类别不平衡问题，以及在困难漏洞检测数据集上取得性能提升。这证明了通过微调大型预训练语言模型实现迁移学习在专业源代码分析任务中的潜力。