Deep neural networks are known to be susceptible to adversarial attacks. In this work, we focus on improving adversarial robustness in the challenging zero-shot image classification setting. To address this issue, we propose LAAT, a novel Language-driven, Anchor-based Adversarial Training strategy. LAAT utilizes a text encoder to generate fixed anchors (normalized feature embeddings) for each category and then uses these anchors for adversarial training. By leveraging the semantic consistency of the text encoders, LAAT can enhance the adversarial robustness of the image model on novel categories without additional examples. We identify the large cosine similarity problem of recent text encoders and design several effective techniques to address it. The experimental results demonstrate that LAAT significantly improves zero-shot adversarial performance, outperforming previous state-of-the-art adversarially robust one-shot methods. Moreover, our method produces substantial zero-shot adversarial robustness when models are trained on large datasets such as ImageNet-1K and applied to several downstream datasets.

翻译：深度神经网络已知易受对抗攻击。本文聚焦于提升具有挑战性的零样本图像分类场景中的对抗鲁棒性。针对该问题，我们提出LAAT（Language-driven, Anchor-based Adversarial Training）——一种新颖的语言驱动、基于锚点的对抗训练策略。LAAT利用文本编码器为每个类别生成固定锚点（归一化特征嵌入），并基于这些锚点进行对抗训练。通过利用文本编码器的语义一致性，LAAT能够无需额外样本即可增强图像模型在新类别上的对抗鲁棒性。我们识别出当前文本编码器存在的余弦相似度过大问题，并设计了多项有效技术予以解决。实验结果表明，LAAT显著提升了零样本对抗性能，超越了先前最先进的对抗鲁棒单样本方法。此外，当模型在大规模数据集（如ImageNet-1K）上训练并应用于多个下游数据集时，本方法展现了显著的零样本对抗鲁棒性。