Deep neural networks (DNNs) have been showed to be highly vulnerable to imperceptible adversarial perturbations. As a complementary type of adversary, patch attacks that introduce perceptible perturbations to the images have attracted the interest of researchers. Existing patch attacks rely on the architecture of the model or the probabilities of predictions and perform poorly in the decision-based setting, which can still construct a perturbation with the minimal information exposed -- the top-1 predicted label. In this work, we first explore the decision-based patch attack. To enhance the attack efficiency, we model the patches using paired key-points and use targeted images as the initialization of patches, and parameter optimizations are all performed on the integer domain. Then, we propose a differential evolutionary algorithm named DevoPatch for query-efficient decision-based patch attacks. Experiments demonstrate that DevoPatch outperforms the state-of-the-art black-box patch attacks in terms of patch area and attack success rate within a given query budget on image classification and face verification. Additionally, we conduct the vulnerability evaluation of ViT and MLP on image classification in the decision-based patch attack setting for the first time. Using DevoPatch, we can evaluate the robustness of models to black-box patch attacks. We believe this method could inspire the design and deployment of robust vision models based on various DNN architectures in the future.

翻译：深度神经网络（DNN）已被证明对不可察觉的对抗性扰动高度脆弱。作为一种补充性对抗类型，引入可察觉扰动的补丁攻击引起了研究者的兴趣。现有补丁攻击依赖模型架构或预测概率，且在仅暴露最少信息（即前1个预测标签）的决策级场景下性能不佳。本文首次探索了决策级补丁攻击。为提升攻击效率，我们采用配对关键点建模补丁，以目标图像作为补丁初始化，并在整数域内进行参数优化。随后，我们提出一种名为DevoPatch的差分进化算法，用于实现查询高效的决策级补丁攻击。实验表明，在图像分类和人脸验证任务中，给定查询预算下，DevoPatch在补丁面积和攻击成功率方面均优于最先进的黑盒补丁攻击方法。此外，我们首次在决策级补丁攻击场景下评估了ViT和MLP模型在图像分类中的脆弱性。借助DevoPatch，可评估模型对黑盒补丁攻击的鲁棒性。我们相信该方法将启发未来基于多种DNN架构的鲁棒视觉模型的设计与部署。