We propose and release a new vulnerable source code dataset. We curate the dataset by crawling security issue websites, extracting vulnerability-fixing commits and source codes from the corresponding projects. Our new dataset contains 18,945 vulnerable functions spanning 150 CWEs and 330,492 non-vulnerable functions extracted from 7,514 commits. Our dataset covers 295 more projects than all previous datasets combined. Combining our new dataset with previous datasets, we present an analysis of the challenges and promising research directions of using deep learning for detecting software vulnerabilities. We study 11 model architectures belonging to 4 families. Our results show that deep learning is still not ready for vulnerability detection, due to high false positive rate, low F1 score, and difficulty of detecting hard CWEs. In particular, we demonstrate an important generalization challenge for the deployment of deep learning-based models. We show that increasing the volume of training data may not further improve the performance of deep learning models for vulnerability detection, but might be useful to improve the generalization ability to unseen projects. We also identify hopeful future research directions. We demonstrate that large language models (LLMs) are a promising research direction for ML-based vulnerability detection, outperforming Graph Neural Networks (GNNs) with code-structure features in our experiments. Moreover, developing source code specific pre-training objectives is a promising research direction to improve the vulnerability detection performance.

翻译：我们提出并发布了一个新的脆弱源代码数据集。通过爬取安全漏洞网站、提取相关项目中的漏洞修复提交及源代码，我们精心构建了该数据集。该新数据集包含来自150个CWE类型的18,945个脆弱函数，以及从7,514个提交中提取的330,492个非脆弱函数。其涵盖的项目数量比以往所有数据集总和多出295个。结合新数据集与现有数据集，我们分析了利用深度学习检测软件漏洞的挑战与潜在研究方向，研究了隶属4个家族的11种模型架构。结果表明，由于高误报率、低F1分数以及难以检测复杂CWE类型，深度学习尚未准备好用于漏洞检测。我们特别揭示了部署深度学习模型面临的重要泛化挑战：增加训练数据量可能无法进一步提升漏洞检测性能，但有助于增强模型对未见项目的泛化能力。同时，我们指出了未来有望取得突破的方向。实验表明，大型语言模型（LLM）在基于机器学习的漏洞检测中具有巨大潜力，其性能优于利用代码结构特征的图神经网络（GNN）。此外，开发针对源代码的预训练目标函数是提升漏洞检测性能的重要研究方向。