Natural language processing models are vulnerable to adversarial examples. Previous textual adversarial attacks adopt gradients or confidence scores to calculate word importance ranking and generate adversarial examples. However, this information is unavailable in the real world. Therefore, we focus on a more realistic and challenging setting, named hard-label attack, in which the attacker can only query the model and obtain a discrete prediction label. Existing hard-label attack algorithms tend to initialize adversarial examples by random substitution and then utilize complex heuristic algorithms to optimize the adversarial perturbation. These methods require a lot of model queries and the attack success rate is restricted by adversary initialization. In this paper, we propose a novel hard-label attack algorithm named LimeAttack, which leverages a local explainable method to approximate word importance ranking, and then adopts beam search to find the optimal solution. Extensive experiments show that LimeAttack achieves the better attacking performance compared with existing hard-label attack under the same query budget. In addition, we evaluate the effectiveness of LimeAttack on large language models, and results indicate that adversarial examples remain a significant threat to large language models. The adversarial examples crafted by LimeAttack are highly transferable and effectively improve model robustness in adversarial training.

翻译：自然语言处理模型易受对抗样本攻击。现有文本对抗攻击方法多依赖梯度或置信度分数计算词重要性排序并生成对抗样本，但实际场景中这些信息难以获取。为此，本文聚焦于更具现实挑战性的硬标签攻击场景——攻击者仅能通过查询模型获取离散预测标签。现有硬标签攻击算法通常采用随机替换初始化对抗样本，再借助复杂启发式算法优化对抗扰动。此类方法需要大量模型查询，且攻击成功率受限于对抗初始化的效果。本文提出新型硬标签攻击算法LimeAttack，通过局部可解释方法近似词重要性排序，并采用束搜索寻找最优解。大量实验表明，在相同查询预算下，LimeAttack相比现有硬标签攻击实现了更优的攻击性能。此外，我们评估了LimeAttack对大型语言模型的有效性，结果表明对抗样本仍对大型语言模型构成显著威胁。LimeAttack生成的对抗样本具有强迁移性，并能有效提升对抗训练中的模型鲁棒性。