The recent success of Vision Transformers is shaking the long dominance of Convolutional Neural Networks (CNNs) in image recognition for a decade. Specifically, in terms of robustness on out-of-distribution samples, recent research finds that Transformers are inherently more robust than CNNs, regardless of different training setups. Moreover, it is believed that such superiority of Transformers should largely be credited to their self-attention-like architectures per se. In this paper, we question that belief by closely examining the design of Transformers. Our findings lead to three highly effective architecture designs for boosting robustness, yet simple enough to be implemented in several lines of code, namely a) patchifying input images, b) enlarging kernel size, and c) reducing activation layers and normalization layers. Bringing these components together, we are able to build pure CNN architectures without any attention-like operations that are as robust as, or even more robust than, Transformers. We hope this work can help the community better understand the design of robust neural architectures. The code is publicly available at https://github.com/UCSC-VLAA/RobustCNN.

翻译：近年来，Vision Transformer的成功正在动摇卷积神经网络（CNN）在图像识别领域长达十年的主导地位。具体而言，在针对分布外样本的鲁棒性方面，最新研究发现无论采用何种训练设置，Transformer本质上都比CNN更鲁棒。此外，研究者普遍认为Transformer的这种优越性应主要归功于其自注意力类架构本身。本文通过深入审视Transformer的设计对这种观点提出质疑。我们的研究发现三种能显著提升鲁棒性且实现极为简洁（仅需数行代码）的高效架构设计：a) 对输入图像进行分块化处理，b) 扩大卷积核尺寸，c) 减少激活层和归一化层。通过整合这些组件，我们能够构建出不含任何注意力机制但鲁棒性可与Transformer媲美甚至更优的纯CNN架构。我们希望这项工作能帮助学界更好地理解鲁棒神经架构的设计原理。代码已开源至 https://github.com/UCSC-VLAA/RobustCNN。