State-of-the-art computer vision models are mostly trained with supervised learning using human-labeled images, which limits their scalability due to the expensive annotation cost. While self-supervised representation learning has achieved impressive progress, it still requires a second stage of finetuning on labeled data. On the other hand, models pre-trained with large-scale text-image supervision (e.g., CLIP) have enabled zero-shot transfer to downstream image classification tasks. However, the zero-shot performance of CLIP-like models are often insufficient for real-world adoption. In this paper, we aim to leverage the abundant unlabeled data from a target domain to improve the performance of a pre-trained zero-shot classifier, by unsupervised finetuning of the pre-trained model. We propose Masked Unsupervised Self-Training (MUST), a new unsupervised adaptation method which leverages two different and complementary sources of training signals: pseudo-labels and raw images. MUST jointly optimizes three objectives to learn both class-level global feature and pixel-level local feature and enforces a regularization between the two. We demonstrate the efficacy of MUST on a variety of downstream tasks, where it improves upon CLIP by a large margin. MUST also outperforms supervised few-shot adaptation methods. It achieves a top-1 accuracy of 77.7% on ImageNet using ViT-B, +9.4% higher than CLIP, and +6.2% higher than 16-shot CLIP adaptation. Our code is available at https://github.com/salesforce/MUST.

翻译：目前最先进的计算机视觉模型大多采用人工标注图像进行监督学习，但由于昂贵的标注成本，其可扩展性受到限制。虽然自监督表示学习已取得显著进展，但仍需在标注数据上进行第二阶段的微调。另一方面，通过大规模图文联合监督预训练的模型（如CLIP）已能实现零样本迁移至下游图像分类任务。然而，CLIP类模型的零样本性能在实际应用中往往不足。本文旨在利用目标领域丰富的无标注数据，通过对预训练模型进行无监督微调来提升预训练零样本分类器的性能。我们提出掩码无监督自训练（MUST），这是一种新型的无监督自适应方法，利用两种互补的训练信号源：伪标签与原始图像。MUST联合优化三个目标函数，同时学习类别级全局特征与像素级局部特征，并在两者间施加正则化约束。我们在多种下游任务中验证了MUST的有效性，其性能显著超越CLIP模型，同时优于监督式少样本自适应方法。基于ViT-B架构，MUST在ImageNet上达到77.7%的top-1准确率，较CLIP提升9.4%，较16样本CLIP自适应提升6.2%。我们的代码开源在https://github.com/salesforce/MUST。