One of the most pressing threats to computing systems is software vulnerabilities, which can compromise both hardware and software components. Existing methods for vulnerability detection remain suboptimal. Traditional techniques are both time-consuming and labor-intensive, while machine-learning-based approaches often underperform when applied to complex datasets, due to their inability to capture high-dimensional relationships. Previous deep-learning strategies also fall short in capturing sufficient feature information. Although self-attention mechanisms can process information over long distances, they fail to capture structural information. In this paper, we introduce DefectHunter, an innovative model for vulnerability identification that employs the Conformer mechanism. This mechanism fuses self-attention with convolutional networks to capture both local, position-wise features and global, content-based interactions. Furthermore, we optimize the self-attention mechanisms to mitigate the issue of excessive attention heads introducing extraneous noise by adjusting the denominator. We evaluated DefectHunter against ten baseline methods using six industrial and two highly complex datasets. On the QEMU dataset, DefectHunter exhibited a 20.62\% improvement in accuracy over Pongo-70B, and for the CWE-754 dataset, its accuracy was 14.64\% higher. To investigate how DefectHunter comprehends vulnerabilities, we conducted a case study, which revealed that our model effectively understands the mechanisms underlying vulnerabilities.

翻译：对计算系统最紧迫的威胁之一是软件漏洞，这类漏洞可能危及硬件和软件组件。现有的漏洞检测方法仍不够理想。传统技术既耗时又费力，而基于机器学习的方法在应用于复杂数据集时往往表现不佳，因无法捕捉高维关系。先前的深度学习策略在提取足够特征信息方面也存在不足。尽管自注意力机制可以处理长距离信息，但无法捕捉结构信息。本文提出DefectHunter，一种采用Conformer机制的创新漏洞识别模型。该机制融合自注意力与卷积网络，既能捕捉局部位置特征，也能捕捉全局基于内容的交互。此外，我们通过调整分母优化了自注意力机制，以缓解注意力头过多引入无关噪声的问题。我们使用六个工业数据集和两个高复杂度数据集，将DefectHunter与十种基线方法进行了评估。在QEMU数据集上，DefectHunter的准确率比Pongo-70B高出20.62%，而在CWE-754数据集上，其准确率提升了14.64%。为探究DefectHunter如何理解漏洞，我们进行了案例研究，结果表明该模型能有效理解漏洞的底层机制。