As software becomes increasingly complex and prone to vulnerabilities, automated vulnerability detection is critically important, yet challenging. Given the significant successes of large language models (LLMs) in various tasks, there is growing anticipation of their efficacy in vulnerability detection. However, a quantitative understanding of their potential in vulnerability detection is still missing. To bridge this gap, we introduce a comprehensive vulnerability benchmark VulBench. This benchmark aggregates high-quality data from a wide range of CTF (Capture-the-Flag) challenges and real-world applications, with annotations for each vulnerable function detailing the vulnerability type and its root cause. Through our experiments encompassing 16 LLMs and 6 state-of-the-art (SOTA) deep learning-based models and static analyzers, we find that several LLMs outperform traditional deep learning approaches in vulnerability detection, revealing an untapped potential in LLMs. This work contributes to the understanding and utilization of LLMs for enhanced software security.

翻译：随着软件日益复杂且易于出现漏洞，自动化漏洞检测至关重要且极具挑战性。鉴于大语言模型（LLMs）在各类任务中取得的显著成功，人们对其在漏洞检测中的有效性寄予厚望。然而，目前仍缺乏对其在该领域潜力的量化理解。为弥补这一空白，我们提出综合性漏洞基准VulBench。该基准整合了来自各类CTF（夺旗赛）挑战及真实应用的高质量数据，并对每个脆弱函数标注漏洞类型及其根源原因。通过涵盖16个大语言模型、6种基于深度学习的最先进模型及静态分析器的实验，我们发现多个大语言模型在漏洞检测中优于传统深度学习方法，揭示了LLMs未被开发的潜力。本研究为增强软件安全性的LLMs理解与应用做出贡献。