Despite the remarkable progress in deep generative models, synthesizing high-resolution and temporally coherent videos still remains a challenge due to their high-dimensionality and complex temporal dynamics along with large spatial variations. Recent works on diffusion models have shown their potential to solve this challenge, yet they suffer from severe computation- and memory-inefficiency that limit the scalability. To handle this issue, we propose a novel generative model for videos, coined projected latent video diffusion models (PVDM), a probabilistic diffusion model which learns a video distribution in a low-dimensional latent space and thus can be efficiently trained with high-resolution videos under limited resources. Specifically, PVDM is composed of two components: (a) an autoencoder that projects a given video as 2D-shaped latent vectors that factorize the complex cubic structure of video pixels and (b) a diffusion model architecture specialized for our new factorized latent space and the training/sampling procedure to synthesize videos of arbitrary length with a single model. Experiments on popular video generation datasets demonstrate the superiority of PVDM compared with previous video synthesis methods; e.g., PVDM obtains the FVD score of 639.7 on the UCF-101 long video (128 frames) generation benchmark, which improves 1773.4 of the prior state-of-the-art.

翻译：尽管深度生成模型取得了显著进展，但由于视频的高维性、复杂的时序动态以及巨大的空间变化，合成高分辨率且时间连贯的视频仍然是一个挑战。近期关于扩散模型的研究表明其具有解决这一挑战的潜力，但这些模型存在严重的计算和内存效率低下问题，限制了可扩展性。为解决这一问题，我们提出了一种新颖的视频生成模型——投影隐空间视频扩散模型（PVDM），这是一种在低维隐空间中学习视频分布的概率扩散模型，因此能够在有限资源下高效训练高分辨率视频。具体而言，PVDM由两个组件组成：（a）一个自编码器，将给定视频投影为二维形状的隐向量，从而分解视频像素的复杂立方体结构；（b）一种专门针对我们新的分解隐空间设计的扩散模型架构，以及训练/采样流程，使单个模型能够合成任意长度的视频。在流行视频生成数据集上的实验证明了PVDM相较于先前视频合成方法的优越性；例如，PVDM在UCF-101长视频（128帧）生成基准上实现了639.7的FVD得分，相较之前的最先进方法提升了1773.4。