Large-scale pre-trained vision-language models like CLIP have demonstrated impressive performance across various tasks, and exhibit remarkable zero-shot generalization capability, while they are also vulnerable to imperceptible adversarial examples. Existing works typically employ adversarial training (fine-tuning) as a defense method against adversarial examples. However, direct application to the CLIP model may result in overfitting, compromising the model's capacity for generalization. In this paper, we propose Pre-trained Model Guided Adversarial Fine-Tuning (PMG-AFT) method, which leverages supervision from the original pre-trained model by carefully designing an auxiliary branch, to enhance the model's zero-shot adversarial robustness. Specifically, PMG-AFT minimizes the distance between the features of adversarial examples in the target model and those in the pre-trained model, aiming to preserve the generalization features already captured by the pre-trained model. Extensive Experiments on 15 zero-shot datasets demonstrate that PMG-AFT significantly outperforms the state-of-the-art method, improving the top-1 robust accuracy by an average of 4.99%. Furthermore, our approach consistently improves clean accuracy by an average of 8.72%. Our code is available at https://github.com/serendipity1122/Pre-trained-Model-Guided-Fine-Tuning-for-Zero-Shot-Adversarial-Robustness.

翻译：大规模预训练视觉语言模型（如CLIP）已在各类任务中展现出卓越性能，并具备显著的零样本泛化能力，但同样容易受到难以察觉的对抗样本攻击。现有方法通常采用对抗训练（微调）作为防御对抗样本的手段，然而直接应用于CLIP模型可能导致过拟合，从而削弱模型的泛化能力。本文提出预训练模型引导的对抗微调方法（PMG-AFT），通过精心设计的辅助分支引入原始预训练模型的监督信号，以增强模型的零样本对抗鲁棒性。具体而言，PMG-AFT通过最小化目标模型中对抗样本特征与预训练模型特征之间的距离，旨在保留预训练模型已捕获的泛化特征。在15个零样本数据集上的大量实验表明，PMG-AFT显著优于当前最优方法，将top-1鲁棒准确率平均提升4.99%。此外，本方法还持续提升干净样本准确率，平均提高8.72%。我们的代码已开源至https://github.com/serendipity1122/Pre-trained-Model-Guided-Fine-Tuning-for-Zero-Shot-Adversarial-Robustness。