We introduce the MAsked Generative VIdeo Transformer, MAGVIT, to tackle various video synthesis tasks with a single model. We introduce a 3D tokenizer to quantize a video into spatial-temporal visual tokens and propose an embedding method for masked video token modeling to facilitate multi-task learning. We conduct extensive experiments to demonstrate the quality, efficiency, and flexibility of MAGVIT. Our experiments show that (i) MAGVIT performs favorably against state-of-the-art approaches and establishes the best-published FVD on three video generation benchmarks, including the challenging Kinetics-600. (ii) MAGVIT outperforms existing methods in inference time by two orders of magnitude against diffusion models and by 60x against autoregressive models. (iii) A single MAGVIT model supports ten diverse generation tasks and generalizes across videos from different visual domains. The source code and trained models will be released to the public at https://magvit.cs.cmu.edu.

翻译：我们提出了掩码生成式视频Transformer（MAsked Generative VIdeo Transformer，MAGVIT），旨在通过单一模型解决多种视频合成任务。我们引入了一种3D分词器，将视频量化为时空视觉标记，并提出了用于掩码视频标记建模的嵌入方法以促进多任务学习。通过大量实验，我们证明了MAGVIT的质量、效率和灵活性。实验表明：(i) MAGVIT在性能上优于最先进方法，并在三个视频生成基准（包括具有挑战性的Kinetics-600）上创下了最佳FVD（Fréchet Video Distance）记录；(ii) MAGVIT的推理时间比扩散模型快两个数量级，比自回归模型快60倍；(iii) 单一MAGVIT模型支持十种不同的生成任务，并可泛化至不同视觉领域的视频。源代码与训练模型将在https://magvit.cs.cmu.edu上公开发布。