This work aims to learn a high-quality text-to-video (T2V) generative model by leveraging a pre-trained text-to-image (T2I) model as a basis. It is a highly desirable yet challenging task to simultaneously a) accomplish the synthesis of visually realistic and temporally coherent videos while b) preserving the strong creative generation nature of the pre-trained T2I model. To this end, we propose LaVie, an integrated video generation framework that operates on cascaded video latent diffusion models, comprising a base T2V model, a temporal interpolation model, and a video super-resolution model. Our key insights are two-fold: 1) We reveal that the incorporation of simple temporal self-attentions, coupled with rotary positional encoding, adequately captures the temporal correlations inherent in video data. 2) Additionally, we validate that the process of joint image-video fine-tuning plays a pivotal role in producing high-quality and creative outcomes. To enhance the performance of LaVie, we contribute a comprehensive and diverse video dataset named Vimeo25M, consisting of 25 million text-video pairs that prioritize quality, diversity, and aesthetic appeal. Extensive experiments demonstrate that LaVie achieves state-of-the-art performance both quantitatively and qualitatively. Furthermore, we showcase the versatility of pre-trained LaVie models in various long video generation and personalized video synthesis applications.

翻译：本工作旨在利用预训练的文本到图像（T2I）模型作为基础，学习一个高质量的文本到视频（T2V）生成模型。同时实现以下两个目标是一项高度期待但具有挑战性的任务：a）生成视觉逼真且时间连贯的视频，b）保留预训练T2I模型的强大创造性生成能力。为此，我们提出LaVie，一个集成视频生成框架，该框架基于级联视频潜扩散模型运行，包括一个基础T2V模型、一个时间插值模型和一个视频超分辨率模型。我们的关键见解有两点：1）我们发现，结合简单的时序自注意力机制与旋转位置编码，能够充分捕捉视频数据中固有的时间相关性。2）此外，我们验证了联合图像-视频微调过程在产生高质量和创造性结果中起着关键作用。为了提升LaVie的性能，我们贡献了一个名为Vimeo25M的综合性多样化视频数据集，包含2500万个优先考虑质量、多样性和美学吸引力的文本-视频对。大量实验表明，LaVie在定量和定性评估上均达到了最先进的性能。此外，我们展示了预训练LaVie模型在各种长视频生成和个性化视频合成应用中的多功能性。