Adversarial attacks, particularly patch attacks, pose significant threats to the robustness and reliability of deep learning models. Developing reliable defenses against patch attacks is crucial for real-world applications, yet current research in this area is unsatisfactory. In this paper, we propose DIFFender, a novel defense method that leverages a text-guided diffusion model to defend against adversarial patches. DIFFender includes two main stages: patch localization and patch restoration. In the localization stage, we find and exploit an intriguing property of the diffusion model to precisely identify the locations of adversarial patches. In the restoration stage, we employ the diffusion model to reconstruct the adversarial regions in the images while preserving the integrity of the visual content. Thanks to the former finding, these two stages can be simultaneously guided by a unified diffusion model. Thus, we can utilize the close interaction between them to improve the whole defense performance. Moreover, we propose a few-shot prompt-tuning algorithm to fine-tune the diffusion model, enabling the pre-trained diffusion model to adapt to the defense task easily. We conduct extensive experiments on image classification, face recognition, and further in the physical world, demonstrating that our proposed method exhibits superior robustness under strong adaptive attacks and generalizes well across various scenarios, diverse classifiers, and multiple patch attack methods.

翻译：对抗性攻击，特别是补丁攻击，对深度学习模型的鲁棒性和可靠性构成严重威胁。开发针对补丁攻击的可靠防御方法对于实际应用至关重要，然而当前该领域的研究仍不尽人意。本文提出DIFFender，一种利用文本引导扩散模型防御对抗性补丁的新型防御方法。DIFFender包含两个主要阶段：补丁定位与补丁恢复。在定位阶段，我们发现并利用扩散模型的一项有趣特性，精准识别对抗性补丁的位置；在恢复阶段，我们采用扩散模型重建图像中的对抗区域，同时保持视觉内容的完整性。得益于前一项发现，这两个阶段可由统一的扩散模型协同引导，从而通过二者的紧密交互提升整体防御性能。此外，我们提出一种少样本提示调优算法对扩散模型进行微调，使预训练扩散模型能够轻松适应防御任务。我们在图像分类、人脸识别以及物理世界场景中进行了广泛实验，结果表明，所提方法在强自适应攻击下展现出卓越的鲁棒性，并能在多种场景、不同分类器及多种补丁攻击方法中实现良好泛化。