Pre-trained language models of code are now widely used in various software engineering tasks such as code generation, code completion, vulnerability detection, etc. This, in turn, poses security and reliability risks to these models. One of the important threats is \textit{adversarial attacks}, which can lead to erroneous predictions and largely affect model performance on downstream tasks. Current adversarial attacks on code models usually adopt fixed sets of program transformations, such as variable renaming and dead code insertion, leading to limited attack effectiveness. To address the aforementioned challenges, we propose a novel adversarial attack framework, GraphCodeAttack, to better evaluate the robustness of code models. Given a target code model, GraphCodeAttack automatically mines important code patterns, which can influence the model's decisions, to perturb the structure of input code to the model. To do so, GraphCodeAttack uses a set of input source codes to probe the model's outputs and identifies the \textit{discriminative} ASTs patterns that can influence the model decisions. GraphCodeAttack then selects appropriate AST patterns, concretizes the selected patterns as attacks, and inserts them as dead code into the model's input program. To effectively synthesize attacks from AST patterns, GraphCodeAttack uses a separate pre-trained code model to fill in the ASTs with concrete code snippets. We evaluate the robustness of two popular code models (e.g., CodeBERT and GraphCodeBERT) against our proposed approach on three tasks: Authorship Attribution, Vulnerability Prediction, and Clone Detection. The experimental results suggest that our proposed approach significantly outperforms state-of-the-art approaches in attacking code models such as CARROT and ALERT.

翻译：代码预训练语言模型现已广泛应用于代码生成、代码补全、漏洞检测等多种软件工程任务。这反过来也给这些模型带来了安全性和可靠性风险。其中一项重要威胁是\textit{对抗攻击}，它可能导致错误的预测，并在很大程度上影响模型在下游任务上的性能。当前针对代码模型的对抗攻击通常采用固定的程序变换集合，例如变量重命名和死代码插入，导致攻击效果有限。为了解决上述挑战，我们提出了一种新颖的对抗攻击框架GraphCodeAttack，以更好地评估代码模型的鲁棒性。给定一个目标代码模型，GraphCodeAttack自动挖掘能够影响模型决策的重要代码模式，以扰动输入到模型的代码结构。为此，GraphCodeAttack使用一组输入源代码来探测模型的输出，并识别能够影响模型决策的\textit{判别性}抽象语法树（AST）模式。随后，GraphCodeAttack选择合适的AST模式，将选定的模式具体化为攻击，并将其作为死代码插入到模型的输入程序中。为了有效地从AST模式合成攻击，GraphCodeAttack使用一个独立的预训练代码模型来用具体的代码片段填充AST。我们在三个任务上评估了两个流行的代码模型（例如CodeBERT和GraphCodeBERT）针对我们提出的方法的鲁棒性：代码归属判定、漏洞预测和克隆检测。实验结果表明，我们提出的方法在攻击代码模型（如CARROT和ALERT）方面显著优于现有最先进的方法。