The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.

翻译：自然语言处理领域近期在基于大规模数据预训练模型方面取得的突破，为计算机视觉领域的基础模型发展铺平了道路。此类模型通过生成通用视觉特征（即无需微调即可跨图像分布和任务适用的特征），可极大简化图像在任何系统中的使用。本研究证明，现有预训练方法（尤其是自监督方法）若在来自多元来源的充足精选数据上进行训练，即可生成此类特征。我们重新审视既有方法，融合不同技术以扩展预训练的数据规模与模型规模。大多数技术贡献聚焦于加速和稳定大规模训练过程。在数据方面，我们提出自动化管线以构建专有、多样且经过甄选的图像数据集，而非采用自监督文献中普遍使用的未精选数据。在模型方面，我们训练了包含10亿参数的ViT模型（Dosovitskiy等人，2020），并通过知识蒸馏将其压缩为一系列更小模型，这些模型在图像级和像素级基准测试中，多数指标均超越现有最优的通用特征模型OpenCLIP（Ilharco等人，2021）。