Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain. Although VideoMAE has trained a robust ViT from limited data, its low-level reconstruction poses convergence difficulties and conflicts with high-level cross-modal alignment. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. To increase data efficiency, we mask out most of the low-semantics video tokens, but selectively align the unmasked tokens with IFM, which serves as the UnMasked Teacher (UMT). By providing semantic guidance, our method enables faster convergence and multimodal friendliness. With a progressive pre-training framework, our model can handle various tasks including scene-related, temporal-related, and complex video-language understanding. Using only public sources for pre-training in 6 days on 32 A100 GPUs, our scratch-built ViT-L/16 achieves state-of-the-art performances on various video tasks. The code and models will be released at https://github.com/OpenGVLab/unmasked_teacher.

翻译：视频基础模型因计算成本高昂和数据稀缺而鲜有深入探索。现有视频基础模型依赖于图像基础模型，但此类模型在向视频领域迁移时面临挑战。尽管VideoMAE能从有限数据中训练出鲁棒ViT，但其低层重建任务会导致收敛困难，并与高层跨模态对齐产生冲突。本文提出一种兼具现有方法优势的训练高效时序敏感视频基础模型方法。为提升数据效率，我们遮蔽语义较弱的视频标记，但选择性地将未遮蔽标记与作为"揭秘教师"(UMT)的图像基础模型对齐。通过提供语义引导，本方法实现了更快的收敛速度与多模态友好性。借助渐进式预训练框架，该模型能处理包括场景理解、时序推理及复杂视频语言理解在内的多种任务。仅使用公开数据源在32块A100 GPU上预训练6天，从头训练的ViT-L/16即在多个视频任务中达到最优性能。代码与模型将于https://github.com/OpenGVLab/unmasked_teacher 开源。