Natural language processing models are vulnerable to adversarial examples. Previous textual adversarial attacks adopt gradients or confidence scores to calculate word importance ranking and generate adversarial examples. However, this information is unavailable in the real world. Therefore, we focus on a more realistic and challenging setting, named hard-label attack, in which the attacker can only query the model and obtain a discrete prediction label. Existing hard-label attack algorithms tend to initialize adversarial examples by random substitution and then utilize complex heuristic algorithms to optimize the adversarial perturbation. These methods require a lot of model queries and the attack success rate is restricted by adversary initialization. In this paper, we propose a novel hard-label attack algorithm named LimeAttack, which leverages a local explainable method to approximate word importance ranking, and then adopts beam search to find the optimal solution. Extensive experiments show that LimeAttack achieves the better attacking performance compared with existing hard-label attack under the same query budget. In addition, we evaluate the effectiveness of LimeAttack on large language models, and results indicate that adversarial examples remain a significant threat to large language models. The adversarial examples crafted by LimeAttack are highly transferable and effectively improve model robustness in adversarial training.

翻译：自然语言处理模型易受对抗样本攻击。现有文本对抗攻击方法通常利用梯度或置信度分数计算词语重要性排序并生成对抗样本。然而，此类信息在真实场景中难以获取。因此，我们关注一种更现实且更具挑战性的设定——硬标签攻击，在此设定中攻击者仅能查询模型并获取离散预测标签。现有硬标签攻击算法通常采用随机替换初始化对抗样本，再通过复杂启发式算法优化对抗扰动。此类方法需要大量模型查询，且攻击成功率受限于对抗初始化。本文提出一种名为LimeAttack的新型硬标签攻击算法，该方法利用局部可解释方法近似词语重要性排序，随后采用束搜索寻找最优解。大量实验表明，在相同查询预算下，LimeAttack相比现有硬标签攻击取得了更优的攻击性能。此外，我们评估了LimeAttack对大型语言模型的有效性，结果表明对抗样本仍对大型语言模型构成显著威胁。由LimeAttack生成的对抗样本具有高迁移性，并能有效提升对抗训练中的模型鲁棒性。