In this paper, we ask whether Vision Transformers (ViTs) can serve as an underlying architecture for improving the adversarial robustness of machine learning models against evasion attacks. While earlier works have focused on improving Convolutional Neural Networks, we show that also ViTs are highly suitable for adversarial training to achieve competitive performance. We achieve this objective using a custom adversarial training recipe, discovered using rigorous ablation studies on a subset of the ImageNet dataset. The canonical training recipe for ViTs recommends strong data augmentation, in part to compensate for the lack of vision inductive bias of attention modules, when compared to convolutions. We show that this recipe achieves suboptimal performance when used for adversarial training. In contrast, we find that omitting all heavy data augmentation, and adding some additional bag-of-tricks ($\varepsilon$-warmup and larger weight decay), significantly boosts the performance of robust ViTs. We show that our recipe generalizes to different classes of ViT architectures and large-scale models on full ImageNet-1k. Additionally, investigating the reasons for the robustness of our models, we show that it is easier to generate strong attacks during training when using our recipe and that this leads to better robustness at test time. Finally, we further study one consequence of adversarial training by proposing a way to quantify the semantic nature of adversarial perturbations and highlight its correlation with the robustness of the model. Overall, we recommend that the community should avoid translating the canonical training recipes in ViTs to robust training and rethink common training choices in the context of adversarial training.

翻译：本文探讨视觉Transformer（ViT）能否作为提升机器学习模型对抗扰动的底层架构。现有工作主要聚焦于改进卷积神经网络，我们证明ViT同样适用于对抗训练以取得具有竞争力的性能。我们通过自定义对抗训练方案实现这一目标，该方案基于对ImageNet数据集子集的严格消融实验发现。经典ViT训练方案建议使用强数据增强，部分原因是为了弥补注意力模块相较于卷积缺乏视觉归纳偏置的缺陷。研究表明该方案用于对抗训练时性能次优。相反，我们发现省略所有重型数据增强并添加额外技巧（ε-热启动和更大权重衰减）可显著提升鲁棒ViT的性能。实验证明本方案可泛化至不同类ViT架构及全量ImageNet-1k大规模模型。此外，通过探究模型鲁棒性的成因，我们证实采用本方案训练时更容易生成强攻击样本，从而在测试阶段获得更强的鲁棒性。最后，我们进一步研究对抗训练的一个后果，提出量化对抗扰动语义性的方法，并揭示其与模型鲁棒性的相关性。总体而言，建议学界避免将经典ViT训练方案直接迁移至鲁棒训练，而应在对抗训练背景下重新审视常见训练选择。