Latent Diffusion Models (LDMs) enable high-quality image synthesis while avoiding excessive compute demands by training a diffusion model in a compressed lower-dimensional latent space. Here, we apply the LDM paradigm to high-resolution video generation, a particularly resource-intensive task. We first pre-train an LDM on images only; then, we turn the image generator into a video generator by introducing a temporal dimension to the latent space diffusion model and fine-tuning on encoded image sequences, i.e., videos. Similarly, we temporally align diffusion model upsamplers, turning them into temporally consistent video super resolution models. We focus on two relevant real-world applications: Simulation of in-the-wild driving data and creative content creation with text-to-video modeling. In particular, we validate our Video LDM on real driving videos of resolution 512 x 1024, achieving state-of-the-art performance. Furthermore, our approach can easily leverage off-the-shelf pre-trained image LDMs, as we only need to train a temporal alignment model in that case. Doing so, we turn the publicly available, state-of-the-art text-to-image LDM Stable Diffusion into an efficient and expressive text-to-video model with resolution up to 1280 x 2048. We show that the temporal layers trained in this way generalize to different fine-tuned text-to-image LDMs. Utilizing this property, we show the first results for personalized text-to-video generation, opening exciting directions for future content creation. Project page: https://research.nvidia.com/labs/toronto-ai/VideoLDM/

翻译：潜变量扩散模型（LDMs）通过在压缩的低维潜空间中训练扩散模型，实现了高质量图像合成，同时避免了过高的计算需求。本文中，我们将LDM范式应用于资源密集型的高分辨率视频生成任务。我们首先仅对图像进行LDM预训练；随后，通过向潜空间扩散模型引入时间维度，并在编码的图像序列（即视频）上进行微调，将图像生成器转化为视频生成器。类似地，我们对扩散模型上采样器进行时间对齐，将其转化为时间一致性的视频超分辨率模型。我们聚焦于两个相关的实际应用：野外驾驶数据的模拟以及基于文本到视频建模的创意内容创作。具体而言，我们基于分辨率为512×1024的真实驾驶视频验证了视频LDM，取得了最先进的性能。此外，我们的方法可轻松利用现成的预训练图像LDM——此情况下仅需训练时间对齐模型。通过这种方式，我们将公开可用的、最先进的文本到图像LDM Stable Diffusion转化为高效且表现力丰富的文本到视频模型，分辨率可达1280×2048。研究表明，以此方式训练的时间层可泛化至不同的微调文本到图像LDM。利用这一特性，我们展示了首个个性化文本到视频生成的结果，为未来内容创作开辟了令人振奋的方向。项目页面：https://research.nvidia.com/labs/toronto-ai/VideoLDM/