This paper presents the results of finetuning large language models (LLMs) for the task of detecting vulnerabilities in source code. We leverage WizardCoder, a recent improvement of the state-of-the-art LLM StarCoder, and adapt it for vulnerability detection through further finetuning. To accelerate training, we modify WizardCoder's training procedure, also we investigate optimal training regimes. For the imbalanced dataset with many more negative examples than positive, we also explore different techniques to improve classification performance. The finetuned WizardCoder model achieves improvement in ROC AUC and F1 measures on balanced and imbalanced vulnerability datasets over CodeBERT-like model, demonstrating the effectiveness of adapting pretrained LLMs for vulnerability detection in source code. The key contributions are finetuning the state-of-the-art code LLM, WizardCoder, increasing its training speed without the performance harm, optimizing the training procedure and regimes, handling class imbalance, and improving performance on difficult vulnerability detection datasets. This demonstrates the potential for transfer learning by finetuning large pretrained language models for specialized source code analysis tasks.

翻译：本文介绍了针对源代码漏洞检测任务的大语言模型微调结果。我们利用WizardCoder（最新改进的顶尖大语言模型StarCoder），通过进一步微调使其适配漏洞检测任务。为加速训练，我们改进了WizardCoder的训练流程，并探究了最优训练方案。针对负样本远多于正样本的数据不平衡问题，我们探索了多种提升分类性能的技术。实验结果表明，微调后的WizardCoder模型在平衡与非平衡漏洞数据集上的ROC AUC和F1指标均优于CodeBERT等基线模型，验证了预训练大语言模型在源代码漏洞检测中的有效性。主要贡献包括：微调顶尖代码大语言模型WizardCoder、在保持性能的前提下提升训练速度、优化训练流程与方案、处理类别不平衡问题，以及在困难漏洞检测数据集上实现性能提升。这展示了通过微调大语言模型实现专用源代码分析任务迁移学习的潜力。