This paper revisits the standard pretrain-then-finetune paradigm used in computer vision for visual recognition tasks. Typically, state-of-the-art foundation models are pretrained using large scale (weakly) supervised datasets with billions of images. We introduce an additional pre-pretraining stage that is simple and uses the self-supervised MAE technique to initialize the model. While MAE has only been shown to scale with the size of models, we find that it scales with the size of the training dataset as well. Thus, our MAE-based pre-pretraining scales with both model and data size making it applicable for training foundation models. Pre-pretraining consistently improves both the model convergence and the downstream transfer performance across a range of model scales (millions to billions of parameters), and dataset sizes (millions to billions of images). We measure the effectiveness of pre-pretraining on 10 different visual recognition tasks spanning image classification, video recognition, object detection, low-shot classification and zero-shot recognition. Our largest model achieves new state-of-the-art results on iNaturalist-18 (91.3%), 1-shot ImageNet-1k (62.1%), and zero-shot transfer on Food-101 (96.0%). Our study reveals that model initialization plays a significant role, even for web-scale pretraining with billions of images.

翻译：本文重新审视了计算机视觉中用于视觉识别任务的标准预训练-微调范式。通常，最先进的基础模型采用数十亿张图像的（弱）监督大规模数据集进行预训练。我们引入了一个简单的预预训练阶段，利用自监督MAE技术对模型进行初始化。虽然MAE此前仅被证明可随模型规模扩展，但我们发现其训练数据集规模同样可扩展。因此，基于MAE的预预训练同时适配模型与数据规模，使其适用于训练基础模型。预预训练能持续提升模型收敛速度与下游迁移性能，涵盖从百万到十亿参数的多类模型规模，以及从百万到十亿图像的多类数据集规模。我们在10项视觉识别任务上衡量了预预训练的有效性，包括图像分类、视频识别、目标检测、小样本分类和零样本识别。最大规模的模型在iNaturalist-18（91.3%）、单样本ImageNet-1k（62.1%）以及Food-101零样本迁移（96.0%）上取得了新的最优结果。本研究表明，即使在拥有数十亿图像的web级预训练中，模型初始化仍发挥着重要作用。