Motion, scene and object are three primary visual components of a video. In particular, objects represent the foreground, scenes represent the background, and motion traces their dynamics. Based on this insight, we propose a two-stage MOtion, Scene and Object decomposition framework (MOSO) for video prediction, consisting of MOSO-VQVAE and MOSO-Transformer. In the first stage, MOSO-VQVAE decomposes a previous video clip into the motion, scene and object components, and represents them as distinct groups of discrete tokens. Then, in the second stage, MOSO-Transformer predicts the object and scene tokens of the subsequent video clip based on the previous tokens and adds dynamic motion at the token level to the generated object and scene tokens. Our framework can be easily extended to unconditional video generation and video frame interpolation tasks. Experimental results demonstrate that our method achieves new state-of-the-art performance on five challenging benchmarks for video prediction and unconditional video generation: BAIR, RoboNet, KTH, KITTI and UCF101. In addition, MOSO can produce realistic videos by combining objects and scenes from different videos.

翻译：运动、场景和对象是视频的三个主要视觉组成部分。其中，对象代表前景，场景代表背景，而运动则刻画其动态变化。基于这一认识，我们提出了一种两阶段的运动、场景和对象分解框架（MOSO），用于视频预测，该框架由MOSO-VQVAE和MOSO-Transformer组成。在第一阶段，MOSO-VQVAE将先前视频片段分解为运动、场景和对象组件，并将其表示为不同的离散标记组。随后在第二阶段，MOSO-Transformer基于先前标记预测后续视频片段的对象和场景标记，并在生成的标记级对象和场景标记中添加动态运动。我们的框架可轻松扩展至无条件视频生成和视频帧插值任务。实验结果表明，我们的方法在BAIR、RoboNet、KTH、KITTI和UCF101五个具有挑战性的视频预测和无条件视频生成基准上取得了新的最优性能。此外，MOSO能够通过组合不同视频中的对象和场景生成逼真视频。