Pre-trained language models of code are now widely used in various software engineering tasks such as code generation, code completion, vulnerability detection, etc. This, in turn, poses security and reliability risks to these models. One of the important threats is \textit{adversarial attacks}, which can lead to erroneous predictions and largely affect model performance on downstream tasks. Current adversarial attacks on code models usually adopt fixed sets of program transformations, such as variable renaming and dead code insertion, leading to limited attack effectiveness. To address the aforementioned challenges, we propose a novel adversarial attack framework, GraphCodeAttack, to better evaluate the robustness of code models. Given a target code model, GraphCodeAttack automatically mines important code patterns, which can influence the model's decisions, to perturb the structure of input code to the model. To do so, GraphCodeAttack uses a set of input source codes to probe the model's outputs and identifies the \textit{discriminative} ASTs patterns that can influence the model decisions. GraphCodeAttack then selects appropriate AST patterns, concretizes the selected patterns as attacks, and inserts them as dead code into the model's input program. To effectively synthesize attacks from AST patterns, GraphCodeAttack uses a separate pre-trained code model to fill in the ASTs with concrete code snippets. We evaluate the robustness of two popular code models (e.g., CodeBERT and GraphCodeBERT) against our proposed approach on three tasks: Authorship Attribution, Vulnerability Prediction, and Clone Detection. The experimental results suggest that our proposed approach significantly outperforms state-of-the-art approaches in attacking code models such as CARROT and ALERT.

翻译：预训练的代码语言模型现已广泛应用于各类软件工程任务中，例如代码生成、代码补全、漏洞检测等。这相应地为这些模型带来了安全性和可靠性风险。其中一项重要威胁是\textit{对抗攻击}，它可能导致错误的预测并严重影响模型在下游任务上的性能。当前针对代码模型的对抗攻击通常采用固定的程序变换集合，如变量重命名和死代码插入，导致攻击效果受限。针对上述挑战，我们提出一种新颖的对抗攻击框架GraphCodeAttack，以更好地评估代码模型的鲁棒性。给定目标代码模型，GraphCodeAttack自动挖掘可影响模型决策的重要代码模式，从而扰动输入代码的结构。为此，GraphCodeAttack利用一组输入源代码探测模型的输出，并识别出能够影响模型决策的\textit{判别性}AST模式。随后，GraphCodeAttack选择合适的AST模式，将这些选定模式具体化为攻击，并将其作为死代码插入到模型的输入程序中。为有效从AST模式合成攻击，GraphCodeAttack使用独立的预训练代码模型，用具体代码片段填充AST。我们在三项任务上评估了两种流行代码模型（例如CodeBERT和GraphCodeBERT）对所提方法的鲁棒性：作者归属、漏洞预测和克隆检测。实验结果表明，我们提出的方法在攻击代码模型（如CARROT和ALERT）方面显著优于现有最先进的方法。