Vision transformers combined with self-supervised learning have enabled the development of models which scale across large datasets for several downstream tasks like classification, segmentation and detection. The low-shot learning capability of these models, across several low-shot downstream tasks, has been largely under explored. We perform a system level study of different self supervised pretext tasks, namely contrastive learning, clustering, and masked image modelling for their low-shot capabilities by comparing the pretrained models. In addition we also study the effects of collapse avoidance methods, namely centring, ME-MAX, sinkhorn, on these downstream tasks. Based on our detailed analysis, we introduce a framework involving both mask image modelling and clustering as pretext tasks, which performs better across all low-shot downstream tasks, including multi-class classification, multi-label classification and semantic segmentation. Furthermore, when testing the model on full scale datasets, we show performance gains in multi-class classification, multi-label classification and semantic segmentation.

翻译：视觉Transformer与自监督学习相结合，使得模型能够在分类、分割和检测等多个下游任务中实现大规模数据集的扩展训练。然而，这些模型在多种少样本下游任务中的低样本学习能力尚未得到充分探索。我们通过比较预训练模型，系统性地研究了不同自监督预训练任务（即对比学习、聚类和掩码图像建模）在低样本场景下的性能表现。此外，我们还分析了避免模型坍缩的方法（包括中心化、ME-MAX和Sinkhorn算法）对这些下游任务的影响。基于详细实验分析，我们提出了一个结合掩码图像建模与聚类作为预训练任务的框架，该框架在包括多类别分类、多标签分类和语义分割在内的所有低样本下游任务中均表现出更优性能。进一步地，当在完整规模数据集上测试时，我们的模型在多类别分类、多标签分类和语义分割任务中均取得了性能提升。