Masked Image Modeling (MIM) methods, like Masked Autoencoders (MAE), efficiently learn a rich representation of the input. However, for adapting to downstream tasks, they require a sufficient amount of labeled data since their rich features code not only objects but also less relevant image background. In contrast, Instance Discrimination (ID) methods focus on objects. In this work, we study how to combine the efficiency and scalability of MIM with the ability of ID to perform downstream classification in the absence of large amounts of labeled data. To this end, we introduce Masked Autoencoder Contrastive Tuning (MAE-CT), a sequential approach that utilizes the implicit clustering of the Nearest Neighbor Contrastive Learning (NNCLR) objective to induce abstraction in the topmost layers of a pre-trained MAE. MAE-CT tunes the rich features such that they form semantic clusters of objects without using any labels. Notably, MAE-CT does not rely on hand-crafted augmentations and frequently achieves its best performances while using only minimal augmentations (crop & flip). Further, MAE-CT is compute efficient as it requires at most 10% overhead compared to MAE re-training. Applied to large and huge Vision Transformer (ViT) models, MAE-CT excels over previous self-supervised methods trained on ImageNet in linear probing, k-NN and low-shot classification accuracy as well as in unsupervised clustering accuracy. With ViT-H/16 MAE-CT achieves a new state-of-the-art in linear probing of 82.2%.

翻译：掩码图像建模（MIM）方法，例如掩码自编码器（MAE），能够高效学习输入的丰富表征。然而，在适应下游任务时，其丰富特征不仅编码了目标对象，还包含了不相关的图像背景，因此需要充足的标注数据。相比之下，实例判别（ID）方法则聚焦于目标对象。在本研究中，我们探讨如何将MIM的高效性与可扩展性，与ID在缺乏大量标注数据情况下进行下游分类的能力相结合。为此，我们提出掩码自编码器对比调优（MAE-CT），这是一种顺序方法，利用最近邻对比学习（NNCLR）目标的隐式聚类，在预训练MAE的顶层网络中诱导抽象化表示。MAE-CT对丰富特征进行调优，使其在不使用任何标签的情况下形成目标的语义聚类。值得注意的是，MAE-CT不依赖手工设计的增强策略，且通常仅需最少的增强（裁剪与翻转）即可达到最佳性能。此外，MAE-CT计算高效，相较于重新训练MAE，其计算开销最多不超过10%。应用于大型及超大型视觉Transformer（ViT）模型时，MAE-CT在ImageNet上的线性探测、k-NN分类、低样本分类准确率以及无监督聚类准确率方面，均超越了先前自监督方法。采用ViT-H/16架构的MAE-CT在线性探测任务中达到了82.2%的最新最优结果。