Despite the remarkable progress in deep generative models, synthesizing high-resolution and temporally coherent videos still remains a challenge due to their high-dimensionality and complex temporal dynamics along with large spatial variations. Recent works on diffusion models have shown their potential to solve this challenge, yet they suffer from severe computation- and memory-inefficiency that limit the scalability. To handle this issue, we propose a novel generative model for videos, coined projected latent video diffusion models (PVDM), a probabilistic diffusion model which learns a video distribution in a low-dimensional latent space and thus can be efficiently trained with high-resolution videos under limited resources. Specifically, PVDM is composed of two components: (a) an autoencoder that projects a given video as 2D-shaped latent vectors that factorize the complex cubic structure of video pixels and (b) a diffusion model architecture specialized for our new factorized latent space and the training/sampling procedure to synthesize videos of arbitrary length with a single model. Experiments on popular video generation datasets demonstrate the superiority of PVDM compared with previous video synthesis methods; e.g., PVDM obtains the FVD score of 639.7 on the UCF-101 long video (128 frames) generation benchmark, which improves 1773.4 of the prior state-of-the-art.

翻译：尽管深度生成模型取得了显著进展，但由于视频的高维度、复杂的时间动态以及大空间变化，合成高分辨率且时间连贯的视频仍然是一项挑战。近期关于扩散模型的研究显示了其解决这一难题的潜力，但这些模型存在严重的计算和内存效率低下问题，限制了其可扩展性。为解决这一问题，我们提出了一种新颖的视频生成模型——投影潜在视频扩散模型（PVDM），这是一种在低维潜在空间中学习视频分布的概率扩散模型，因此能够在有限资源下高效训练高分辨率视频。具体而言，PVDM由两个组件构成：（a）一个自编码器，将给定视频投影为二维形状的潜在向量，从而分解视频像素的复杂立方体结构；（b）一个专门针对我们新分解的潜在空间设计的扩散模型架构，以及训练/采样流程，能够使用单一模型合成任意长度的视频。在常用视频生成数据集上的实验表明，PVDM相较于以往的视频合成方法具有优越性；例如，在UCF-101长视频（128帧）生成基准测试中，PVDM获得了639.7的FVD分数，相比此前最先进方法的1773.4有了显著提升。