With the increasing release of powerful language models trained on large code corpus (e.g. CodeBERT was trained on 6.4 million programs), a new family of mutation testing tools has arisen with the promise to generate more "natural" mutants in the sense that the mutated code aims at following the implicit rules and coding conventions typically produced by programmers. In this paper, we study to what extent the mutants produced by language models can semantically mimic the observable behavior of security-related vulnerabilities (a.k.a. Vulnerability-mimicking Mutants), so that designing test cases that are failed by these mutants will help in tackling mimicked vulnerabilities. Since analyzing and running mutants is computationally expensive, it is important to prioritize those mutants that are more likely to be vulnerability mimicking prior to any analysis or test execution. Taking this into account, we introduce VMMS, a machine learning based approach that automatically extracts the features from mutants and predicts the ones that mimic vulnerabilities. We conducted our experiments on a dataset of 45 vulnerabilities and found that 16.6% of the mutants fail one or more tests that are failed by 88.9% of the respective vulnerabilities. More precisely, 3.9% of the mutants from the entire mutant set are vulnerability-mimicking mutants that mimic 55.6% of the vulnerabilities. Despite the scarcity, VMMS predicts vulnerability-mimicking mutants with 0.63 MCC, 0.80 Precision, and 0.51 Recall, demonstrating that the features of vulnerability-mimicking mutants can be automatically learned by machine learning models to statically predict these without the need of investing effort in defining such features.

翻译：随着基于大型代码语料库训练的强大语言模型（如CodeBERT在640万个程序上训练）的不断发布，新一代变异测试工具应运而生，其承诺能生成更"自然"的变异体——即变异代码旨在遵循程序员通常产生的隐式规则和编码惯例。本文研究语言模型生成的变异体在多大程度上能在语义上模仿安全相关漏洞的可观察行为（即漏洞模仿变异体），从而使设计能使这些变异体失败的测试用例有助于应对被模仿的漏洞。由于分析和运行变异体的计算成本高昂，因此在任何分析或测试执行之前优先考虑那些更可能模仿漏洞的变异体至关重要。针对这一点，我们提出了VMMS——一种基于机器学习的方法，可自动从变异体中提取特征并预测哪些变异体模仿漏洞。我们在包含45个漏洞的数据集上开展实验，发现16.6%的变异体无法通过一个或多个测试，而这些测试分别被88.9%的对应漏洞所失败。更精确地说，整个变异体集合中3.9%的变异体属于漏洞模仿变异体，它们模仿了55.6%的漏洞。尽管数量稀少，VMMS预测漏洞模仿变异体的MCC值达0.63，精确率达0.80，召回率达0.51，这表明漏洞模仿变异体的特征可被机器学习模型自动学习，从而无需投入精力定义这些特征即可进行静态预测。