Latent Diffusion Models (LDMs) enable high-quality image synthesis while avoiding excessive compute demands by training a diffusion model in a compressed lower-dimensional latent space. Here, we apply the LDM paradigm to high-resolution video generation, a particularly resource-intensive task. We first pre-train an LDM on images only; then, we turn the image generator into a video generator by introducing a temporal dimension to the latent space diffusion model and fine-tuning on encoded image sequences, i.e., videos. Similarly, we temporally align diffusion model upsamplers, turning them into temporally consistent video super resolution models. We focus on two relevant real-world applications: Simulation of in-the-wild driving data and creative content creation with text-to-video modeling. In particular, we validate our Video LDM on real driving videos of resolution 512 x 1024, achieving state-of-the-art performance. Furthermore, our approach can easily leverage off-the-shelf pre-trained image LDMs, as we only need to train a temporal alignment model in that case. Doing so, we turn the publicly available, state-of-the-art text-to-image LDM Stable Diffusion into an efficient and expressive text-to-video model with resolution up to 1280 x 2048. We show that the temporal layers trained in this way generalize to different fine-tuned text-to-image LDMs. Utilizing this property, we show the first results for personalized text-to-video generation, opening exciting directions for future content creation. Project page: https://research.nvidia.com/labs/toronto-ai/VideoLDM/

翻译：潜变量扩散模型（LDMs）通过在压缩的低维潜空间中训练扩散模型，在避免过度计算需求的同时实现高质量图像合成。本文将此LDM范式应用于高分辨率视频生成——一项资源密集型任务。我们首先仅在图像上预训练LDM；随后，通过向潜空间扩散模型引入时间维度，并在编码后的图像序列（即视频）上微调，将图像生成器转化为视频生成器。类似地，我们对扩散模型上采样器进行时间对齐，将其转化为时间一致的视频超分辨率模型。我们聚焦于两个实际应用场景：野外驾驶数据的模拟与基于文本到视频建模的创意内容生成。具体而言，我们在分辨率为512×1024的真实驾驶视频上验证了视频LDM，取得了当前最优性能。此外，我们的方法可轻松利用现成的预训练图像LDM——在此情况下仅需训练一个时间对齐模型。由此，我们将公开可用的当代最优文本到图像LDM Stable Diffusion转化为高效且富有表现力的文本到视频模型，分辨率最高可达1280×2048。实验表明，以这种方式训练的时间层可泛化至不同微调后的文本到图像LDM。利用这一特性，我们首次展示了个性化文本到视频生成的结果，为未来内容创作开辟了激动人心的方向。项目页面：https://research.nvidia.com/labs/toronto-ai/VideoLDM/