Adversarial examples are important to test and enhance the robustness of deep code models. As source code is discrete and has to strictly stick to complex grammar and semantics constraints, the adversarial example generation techniques in other domains are hardly applicable. Moreover, the adversarial example generation techniques specific to deep code models still suffer from unsatisfactory effectiveness due to the enormous ingredient search space. In this work, we propose a novel adversarial example generation technique (i.e., CODA) for testing deep code models. Its key idea is to use code differences between the target input (i.e., a given code snippet as the model input) and reference inputs (i.e., the inputs that have small code differences but different prediction results with the target input) to guide the generation of adversarial examples. It considers both structure differences and identifier differences to preserve the original semantics. Hence, the ingredient search space can be largely reduced as the one constituted by the two kinds of code differences, and thus the testing process can be improved by designing and guiding corresponding equivalent structure transformations and identifier renaming transformations. Our experiments on 15 deep code models demonstrate the effectiveness and efficiency of CODA, the naturalness of its generated examples, and its capability of enhancing model robustness after adversarial fine-tuning. For example, CODA reveals 88.05% and 72.51% more faults in models than the state-of-the-art techniques (i.e., CARROT and ALERT) on average, respectively.

翻译：对抗样本对于测试和增强深度代码模型的鲁棒性具有重要意义。由于源代码具有离散性且必须严格遵循复杂的语法和语义约束，其他领域的对抗样本生成技术难以直接适用。此外，针对深度代码模型的对抗样本生成技术因成分搜索空间巨大，仍存在有效性不足的问题。本文提出一种新型对抗样本生成技术（即CODA）用于测试深度代码模型，其核心思想是利用目标输入（即作为模型输入的给定代码片段）与参考输入（即与目标输入代码差异小但预测结果不同的输入）之间的代码差异来指导对抗样本的生成。该方法同时考虑结构差异和标识符差异以保留原始语义，从而将成分搜索空间大幅缩减为由这两类代码差异构成的空间。通过设计并引导相应的等价结构变换和标识符重命名变换，测试过程得以显著优化。我们在15个深度代码模型上的实验表明，CODA在有效性、效率、生成样本的自然性以及通过对抗微调增强模型鲁棒性方面均表现优异。例如，CODA相较于最优技术（CARROT和ALERT）平均分别多揭示88.05%和72.51%的模型缺陷。