Deep learning (DL) models of code have recently reported great progress for vulnerability detection. In some cases, DL-based models have outperformed static analysis tools. Although many great models have been proposed, we do not yet have a good understanding of these models. This limits the further advancement of model robustness, debugging, and deployment for the vulnerability detection. In this paper, we surveyed and reproduced 9 state-of-the-art (SOTA) deep learning models on 2 widely used vulnerability detection datasets: Devign and MSR. We investigated 6 research questions in three areas, namely model capabilities, training data, and model interpretation. We experimentally demonstrated the variability between different runs of a model and the low agreement among different models' outputs. We investigated models trained for specific types of vulnerabilities compared to a model that is trained on all the vulnerabilities at once. We explored the types of programs DL may consider "hard" to handle. We investigated the relations of training data sizes and training data composition with model performance. Finally, we studied model interpretations and analyzed important features that the models used to make predictions. We believe that our findings can help better understand model results, provide guidance on preparing training data, and improve the robustness of the models. All of our datasets, code, and results are available at https://doi.org/10.6084/m9.figshare.20791240.

翻译：深度学习（DL）代码模型近期在漏洞检测领域取得了显著进展。在某些情况下，基于深度学习的方法已超越静态分析工具的表现。尽管已有大量优秀模型被提出，但我们对这些模型的理解仍然有限，这制约了漏洞检测领域模型鲁棒性提升、调试优化及部署应用的进一步发展。本文对两个广泛使用的漏洞检测数据集（Devign和MSR）上的9种经典深度学习（SOTA）模型进行了系统调研与复现。我们从模型能力、训练数据与模型解释三个维度开展了6项研究问题，通过实验揭示了模型不同运行间的变异性以及不同模型输出间的低一致性。我们对比了针对特定漏洞类型训练的模型与统一训练的全局模型，探究了深度学习可能认为“难”处理的程序类型，分析了训练数据规模与构成对模型性能的影响。最后，我们研究了模型的可解释性，解析了模型用于决策的关键特征。我们认为本研究有助于深入理解模型预测结果，为训练数据准备提供指导，并提升模型鲁棒性。所有数据集、代码及实验数据均可在https://doi.org/10.6084/m9.figshare.20791240 获取。