Adversarial training has been demonstrated to be one of the most effective remedies for defending adversarial examples, yet it often suffers from the huge robustness generalization gap on unseen testing adversaries, deemed as the adversarially robust generalization problem. Despite the preliminary understandings devoted to adversarially robust generalization, little is known from the architectural perspective. To bridge the gap, this paper for the first time systematically investigated the relationship between adversarially robust generalization and architectural design. Inparticular, we comprehensively evaluated 20 most representative adversarially trained architectures on ImageNette and CIFAR-10 datasets towards multiple `p-norm adversarial attacks. Based on the extensive experiments, we found that, under aligned settings, Vision Transformers (e.g., PVT, CoAtNet) often yield better adversarially robust generalization while CNNs tend to overfit on specific attacks and fail to generalize on multiple adversaries. To better understand the nature behind it, we conduct theoretical analysis via the lens of Rademacher complexity. We revealed the fact that the higher weight sparsity contributes significantly towards the better adversarially robust generalization of Transformers, which can be often achieved by the specially-designed attention blocks. We hope our paper could help to better understand the mechanism for designing robust DNNs. Our model weights can be found at http://robust.art.

翻译：对抗训练已被证明是防御对抗样本最有效的方法之一，但它在面对未见过的测试对抗样本时常常存在巨大的鲁棒性泛化差距，这被称为对抗鲁棒泛化问题。尽管已有初步研究致力于理解对抗鲁棒泛化，但从架构角度的研究仍十分缺乏。为填补这一空白，本文首次系统地研究了对抗鲁棒泛化与架构设计之间的关系。具体而言，我们在ImageNette和CIFAR-10数据集上全面评估了20种最具代表性的经过对抗训练的架构，针对多种`p范数对抗攻击进行了测试。基于大量实验，我们发现在对齐设置下，视觉变换器（如PVT、CoAtNet）通常能实现更好的对抗鲁棒泛化，而卷积神经网络往往在特定攻击上过拟合，难以对多种攻击进行泛化。为深入理解其本质，我们通过Rademacher复杂度的视角进行了理论分析。我们揭示了一个事实：更高的权重稀疏性对变换器更好的对抗鲁棒泛化贡献显著，而这通常可以通过特别设计的注意力模块实现。希望我们的论文有助于更好地理解设计鲁棒深度神经网络的机制。我们的模型权重可在http://robust.art获取。