Generating a video given the first several static frames is challenging as it anticipates reasonable future frames with temporal coherence. Besides video prediction, the ability to rewind from the last frame or infilling between the head and tail is also crucial, but they have rarely been explored for video completion. Since there could be different outcomes from the hints of just a few frames, a system that can follow natural language to perform video completion may significantly improve controllability. Inspired by this, we introduce a novel task, text-guided video completion (TVC), which requests the model to generate a video from partial frames guided by an instruction. We then propose Multimodal Masked Video Generation (MMVG) to address this TVC task. During training, MMVG discretizes the video frames into visual tokens and masks most of them to perform video completion from any time point. At inference time, a single MMVG model can address all 3 cases of TVC, including video prediction, rewind, and infilling, by applying corresponding masking conditions. We evaluate MMVG in various video scenarios, including egocentric, animation, and gaming. Extensive experimental results indicate that MMVG is effective in generating high-quality visual appearances with text guidance for TVC.

翻译：给定前几帧静态图像生成视频具有挑战性，因为它需要预测具有时间连贯性的合理未来帧。除视频预测外，从最后一帧回溯或在首尾帧之间进行插值的能力同样至关重要，但这些在视频补全任务中鲜有探索。由于仅凭少量帧的线索可能产生多种不同的结果，能够遵循自然语言指令进行视频补全的系统将显著提升可控性。基于此，我们提出一项新任务——文本引导视频补全（TVC），要求模型根据指令从部分帧生成完整视频。为解决该任务，我们进一步提出多模态掩码视频生成（MMVG）方法。训练时，MMVG将视频帧离散化为视觉标记，并通过掩码大部分标记实现从任意时间点的视频补全。推理时，单个MMVG模型可通过应用对应掩码条件处理TVC的全部三种情况：视频预测、回溯与插值。我们在包括自我中心视角、动画与游戏在内的多种视频场景中评估了MMVG。大量实验结果表明，MMVG能在文本引导下为TVC生成高质量视觉内容。