This work aims to learn a high-quality text-to-video (T2V) generative model by leveraging a pre-trained text-to-image (T2I) model as a basis. It is a highly desirable yet challenging task to simultaneously a) accomplish the synthesis of visually realistic and temporally coherent videos while b) preserving the strong creative generation nature of the pre-trained T2I model. To this end, we propose LaVie, an integrated video generation framework that operates on cascaded video latent diffusion models, comprising a base T2V model, a temporal interpolation model, and a video super-resolution model. Our key insights are two-fold: 1) We reveal that the incorporation of simple temporal self-attentions, coupled with rotary positional encoding, adequately captures the temporal correlations inherent in video data. 2) Additionally, we validate that the process of joint image-video fine-tuning plays a pivotal role in producing high-quality and creative outcomes. To enhance the performance of LaVie, we contribute a comprehensive and diverse video dataset named Vimeo25M, consisting of 25 million text-video pairs that prioritize quality, diversity, and aesthetic appeal. Extensive experiments demonstrate that LaVie achieves state-of-the-art performance both quantitatively and qualitatively. Furthermore, we showcase the versatility of pre-trained LaVie models in various long video generation and personalized video synthesis applications.

翻译：本工作旨在利用预训练的文生图（T2I）模型作为基础，学习高质量的文生视频（T2V）生成模型。同时实现以下两个目标是一项高度理想但具有挑战性的任务：（a）合成视觉上真实且时间上连贯的视频；（b）保留预训练文生图模型强大的创造性生成能力。为此，我们提出LaVie，一个基于级联视频潜扩散模型的集成视频生成框架，包含基础文生视频模型、时间插值模型和视频超分辨率模型。我们的核心见解有两方面：1）我们发现，引入简单的时间自注意力机制结合旋转位置编码，能够充分捕捉视频数据中固有的时间相关性。2）此外，我们验证了联合图像-视频微调过程对于生成高质量且富有创造性成果的关键作用。为提升LaVie性能，我们贡献了一个名为Vimeo25M的综合多样化视频数据集，包含2500万个以质量、多样性和美学吸引力为优先标准的文本-视频对。广泛实验表明，LaVie在定量和定性指标上均达到了最先进水平。此外，我们展示了预训练LaVie模型在长视频生成和个性化视频合成等多样化应用中的通用性。