A diffusion probabilistic model (DPM), which constructs a forward diffusion process by gradually adding noise to data points and learns the reverse denoising process to generate new samples, has been shown to handle complex data distribution. Despite its recent success in image synthesis, applying DPMs to video generation is still challenging due to high-dimensional data spaces. Previous methods usually adopt a standard diffusion process, where frames in the same video clip are destroyed with independent noises, ignoring the content redundancy and temporal correlation. This work presents a decomposed diffusion process via resolving the per-frame noise into a base noise that is shared among all frames and a residual noise that varies along the time axis. The denoising pipeline employs two jointly-learned networks to match the noise decomposition accordingly. Experiments on various datasets confirm that our approach, termed as VideoFusion, surpasses both GAN-based and diffusion-based alternatives in high-quality video generation. We further show that our decomposed formulation can benefit from pre-trained image diffusion models and well-support text-conditioned video creation.

翻译：扩散概率模型（DPM）通过逐步向数据点添加噪声构建前向扩散过程，并学习反向去噪过程以生成新样本，已被证明能够处理复杂的数据分布。尽管其在图像合成领域取得了近期成功，但由于高维数据空间的存在，将DPM应用于视频生成仍具挑战性。以往方法通常采用标准扩散过程，即在同一视频片段中引入独立噪声破坏各帧，忽略了内容的冗余性和时间相关性。本文提出一种分解扩散过程，通过将逐帧噪声分解为所有帧共享的基底噪声和随时间轴变化的残差噪声来实现。其去噪流程采用两个联合学习的网络以匹配相应的噪声分解。在多种数据集上的实验表明，我们提出的方法VideoFusion在高质量视频生成方面超越了基于生成对抗网络和基于扩散模型的替代方案。我们进一步证明，这种分解式框架能够受益于预训练的图像扩散模型，并有效支持文本条件的视频创作。