Software vulnerabilities can have serious consequences, which is why many techniques have been proposed to defend against them. Among these, vulnerability detection techniques are a major area of focus. However, there is a lack of a comprehensive approach for benchmarking these proposed techniques. In this paper, we present the first survey that comprehensively investigates and summarizes the current state of software vulnerability detection benchmarking. We review the current literature on benchmarking vulnerability detection, including benchmarking approaches in technique-proposing papers and empirical studies. We also separately discuss the benchmarking approaches for traditional and deep learning-based vulnerability detection techniques. Our survey analyzes the challenges of benchmarking software vulnerability detection techniques and the difficulties involved. We summarize the challenges of benchmarking software vulnerability detection techniques and describe possible solutions for addressing these challenges.

翻译：软件漏洞可能导致严重后果，因而众多防御技术被提出。其中，漏洞检测技术是重点研究方向。然而，目前缺乏针对这些技术进行综合基准测评的体系化方法。本文首次全面调研并总结了软件漏洞检测基准测评的现状。我们回顾了当前关于漏洞检测基准测评的文献，涵盖技术论文中的基准测评方法及实证研究。同时，分别讨论了传统与基于深度学习的漏洞检测技术的基准测评方法。本研究分析了软件漏洞检测技术基准测评面临的挑战与难点，总结了相关难题，并阐述了应对这些挑战的可行方案。