Scale is the primary factor for building a powerful foundation model that could well generalize to a variety of downstream tasks. However, it is still challenging to train video foundation models with billions of parameters. This paper shows that video masked autoencoder (VideoMAE) is a scalable and general self-supervised pre-trainer for building video foundation models. We scale the VideoMAE in both model and data with a core design. Specifically, we present a dual masking strategy for efficient pre-training, with an encoder operating on a subset of video tokens and a decoder processing another subset of video tokens. Although VideoMAE is very efficient due to high masking ratio in encoder, masking decoder can still further reduce the overall computational cost. This enables the efficient pre-training of billion-level models in video. We also use a progressive training paradigm that involves an initial pre-training on a diverse multi-sourced unlabeled dataset, followed by a post-pre-training on a mixed labeled dataset. Finally, we successfully train a video ViT model with a billion parameters, which achieves a new state-of-the-art performance on the datasets of Kinetics (90.0% on K400 and 89.9% on K600) and Something-Something (68.7% on V1 and 77.0% on V2). In addition, we extensively verify the pre-trained video ViT models on a variety of downstream tasks, demonstrating its effectiveness as a general video representation learner.

翻译：规模是构建能够良好泛化至多种下游任务的强大基础模型的主要因素。然而，训练具有数十亿参数的视频基础模型仍具挑战性。本文证明视频掩码自编码器（VideoMAE）是一种可扩展且通用的自监督预训练器，适用于构建视频基础模型。我们通过核心设计在模型和数据两个维度上对VideoMAE进行缩放。具体而言，我们提出了一种双重掩码策略用于高效预训练：编码器处理视频令牌子集，解码器处理另一视频令牌子集。尽管VideoMAE因编码器的高掩码率而非常高效，但解码器掩码仍能进一步降低整体计算成本。这使得视频领域十亿级模型的高效预训练成为可能。我们还采用渐进式训练范式，先在多样化多源无标签数据集上进行初始预训练，随后在混合标签数据集上进行后预训练。最终，我们成功训练了一个包含十亿参数的视频ViT模型，在Kinetics数据集（K400达90.0%，K600达89.9%）和Something-Something数据集（V1达68.7%，V2达77.0%）上均取得了新的最佳性能。此外，我们广泛验证了预训练视频ViT模型在多种下游任务中的表现，证明了其作为通用视频表示学习器的有效性。