Vision Transformers have been rapidly uprising in computer vision thanks to their outstanding scaling trends, and gradually replacing convolutional neural networks (CNNs). Recent works on self-supervised learning (SSL) introduce siamese pre-training tasks, on which Transformer backbones continue to demonstrate ever stronger results than CNNs. People come to believe that Transformers or self-attention modules are inherently more suitable than CNNs in the context of SSL. However, it is noteworthy that most if not all prior arts of SSL with CNNs chose the standard ResNets as their backbones, whose architecture effectiveness is known to already lag behind advanced Vision Transformers. Therefore, it remains unclear whether the self-attention operation is crucial for the recent advances in SSL - or CNNs can deliver the same excellence with more advanced designs, too? Can we close the SSL performance gap between Transformers and CNNs? To answer these intriguing questions, we apply self-supervised pre-training to the recently proposed, stronger lager-kernel CNN architecture and conduct an apple-to-apple comparison with Transformers, in their SSL performance. Our results show that we are able to build pure CNN SSL architectures that perform on par with or better than the best SSL-trained Transformers, by just scaling up convolutional kernel sizes besides other small tweaks. Impressively, when transferring to the downstream tasks \texttt{MS COCO} detection and segmentation, our SSL pre-trained CNN model (trained in 100 epochs) achieves the same good performance as the 300-epoch pre-trained Transformer counterpart. We hope this work can help to better understand what is essential (or not) for self-supervised learning backbones.

翻译：视觉Transformer因其卓越的扩展趋势在计算机视觉领域迅速崛起，并逐渐取代卷积神经网络（CNN）。近期关于自监督学习（SSL）的研究引入孪生预训练任务，在此类任务中，Transformer骨干网络持续展现出比CNN更强的性能。人们逐渐相信，在SSL背景下，Transformer或自注意力模块本质上比CNN更具优势。然而值得注意的是，绝大多数（若非全部）基于CNN的SSL先前研究均选用标准ResNet作为骨干网络，而这类架构的有效性已知已落后于先进的视觉Transformer。因此，自注意力操作对于SSL的最新进展是否关键——抑或CNN通过更先进的设计也能达到同样优异的表现——仍属未知。我们能否弥合Transformer与CNN在SSL性能上的差距？为解答这些引人深思的问题，我们将自监督预训练应用于近期提出的更强的大核CNN架构，并与其Transformer在SSL性能上进行公平比较。结果表明，通过扩大卷积核大小并辅以其他微调手段，我们能够构建出与最优SSL训练Transformer表现相当甚至更优的纯CNN SSL架构。令人印象深刻的是，在迁移至下游任务\texttt{MS COCO}检测与分割时，我们的SSL预训练CNN模型（训练100个epoch）能达到与预训练300个epoch的Transformer对应模型相同的优异性能。我们希望这项工作有助于更深入理解自监督学习骨干网络的核心要素（或非核心要素）。