We present a comparative study on how and why contrastive learning (CL) and masked image modeling (MIM) differ in their representations and in their performance of downstream tasks. In particular, we demonstrate that self-supervised Vision Transformers (ViTs) have the following properties: (1) CL trains self-attentions to capture longer-range global patterns than MIM, such as the shape of an object, especially in the later layers of the ViT architecture. This CL property helps ViTs linearly separate images in their representation spaces. However, it also makes the self-attentions collapse into homogeneity for all query tokens and heads. Such homogeneity of self-attention reduces the diversity of representations, worsening scalability and dense prediction performance. (2) CL utilizes the low-frequency signals of the representations, but MIM utilizes high-frequencies. Since low- and high-frequency information respectively represent shapes and textures, CL is more shape-oriented and MIM more texture-oriented. (3) CL plays a crucial role in the later layers, while MIM mainly focuses on the early layers. Upon these analyses, we find that CL and MIM can complement each other and observe that even the simplest harmonization can help leverage the advantages of both methods. The code is available at https://github.com/naver-ai/cl-vs-mim.

翻译：我们提出了对比学习（CL）与掩码图像建模（MIM）在表征学习及下游任务性能方面差异的对比研究。具体而言，我们证明了自监督视觉Transformer（ViT）具有以下特性：（1）对比学习训练的自注意力机制能捕获比掩码图像建模更长的全局模式（如物体形状），尤其在ViT架构的后期层中。这一特性有助于ViT在表征空间中对图像进行线性分离，但同时也导致所有查询令牌和注意力头的自注意力趋于同质性。这种自注意力的同质性降低了表征多样性，损害了可扩展性与密集预测性能。（2）对比学习利用表征中的低频信号，而掩码图像建模利用高频信号。由于低频信息对应形状、高频信息对应纹理，对比学习更侧重形状特征，而掩码图像建模更侧重纹理特征。（3）对比学习在后期层发挥关键作用，而掩码图像建模主要作用于早期层。基于上述分析，我们发现对比学习与掩码图像建模可以相互补充，即使最简单的协调方法也能融合两者的优势。代码已开源：https://github.com/naver-ai/cl-vs-mim。