Masked autoencoding has shown excellent performance on self-supervised video representation learning. Temporal redundancy has led to a high masking ratio and customized masking strategy in VideoMAE. In this paper, we aim to further improve the performance of video masked autoencoding by introducing a motion guided masking strategy. Our key insight is that motion is a general and unique prior in video, which should be taken into account during masked pre-training. Our motion guided masking explicitly incorporates motion information to build temporal consistent masking volume. Based on this masking volume, we can track the unmasked tokens in time and sample a set of temporal consistent cubes from videos. These temporal aligned unmasked tokens will further relieve the information leakage issue in time and encourage the MGMAE to learn more useful structure information. We implement our MGMAE with an online efficient optical flow estimator and backward masking map warping strategy. We perform experiments on the datasets of Something-Something V2 and Kinetics-400, demonstrating the superior performance of our MGMAE to the original VideoMAE. In addition, we provide the visualization analysis to illustrate that our MGMAE can sample temporal consistent cubes in a motion-adaptive manner for more effective video pre-training.

翻译：掩码自编码在自监督视频表示学习中展现出卓越性能。时间冗余导致VideoMAE中采用了高掩码比例和定制化掩码策略。本文旨在通过引入运动引导掩码策略进一步提升视频掩码自编码的性能。关键见解在于，运动作为视频中普遍且独特的先验信息，应在掩码预训练过程中被充分考虑。我们的运动引导掩码显式地融合运动信息，构建时序一致的掩码体。基于该掩码体，可对未掩码标记进行时间追踪，并从视频中采样一组时序一致的立方体。这些时序对齐的未掩码标记将进一步缓解时间维度的信息泄露问题，促使MGMAE学习更有价值的结构信息。我们通过在线高效光流估计器与后向掩码图扭曲策略实现MGMAE。在Something-Something V2和Kinetics-400数据集上的实验表明，MGMAE相较于原始VideoMAE具有更优性能。此外，可视化分析验证了MGMAE能以运动自适应方式采样时序一致立方体，实现更高效的视频预训练。