Deep neural networks are vulnerable to adversarial attacks. We consider adversarial defense in the case of zero-shot image classification setting, which has rarely been explored because both adversarial defense and zero-shot learning are challenging. We propose LAAT, a novel Language-driven, Anchor-based Adversarial Training strategy, to improve the adversarial robustness in a zero-shot setting. LAAT uses a text encoder to obtain fixed anchors (normalized feature embeddings) of each category, then uses these anchors to perform adversarial training. The text encoder has the property that semantically similar categories can be mapped to neighboring anchors in the feature space. By leveraging this property, LAAT can make the image model adversarially robust on novel categories without any extra examples. Experimental results show that our method achieves impressive zero-shot adversarial performance, even surpassing the previous state-of-the-art adversarially robust one-shot methods in most attacking settings. When models are trained with LAAT on large datasets like ImageNet-1K, they can have substantial zero-shot adversarial robustness across several downstream datasets.

翻译：深度神经网络容易受到对抗攻击的影响。我们考虑在零样本图像分类设置下的对抗防御问题，由于对抗防御和零样本学习都具有挑战性，这一方向鲜有探索。我们提出LAAT——一种新颖的语言驱动、基于锚点的对抗训练策略，旨在提升零样本场景下的对抗鲁棒性。LAAT利用文本编码器获取每个类别的固定锚点（归一化特征嵌入），然后使用这些锚点进行对抗训练。文本编码器具有一个特性：语义相似的类别在特征空间中被映射到邻近的锚点。借助这一特性，LAAT无需任何额外样本即可使图像模型对新型类别具有对抗鲁棒性。实验结果表明，我们的方法在零样本对抗性能上取得了令人瞩目的成果，甚至在多数攻击设置下超越了之前最先进的单样本对抗鲁棒方法。当在ImageNet-1K等大型数据集上使用LAAT训练模型时，模型在多个下游数据集上展现出显著的零样本对抗鲁棒性。