Despite tremendous progress in generating high-quality images using diffusion models, synthesizing a sequence of animated frames that are both photorealistic and temporally coherent is still in its infancy. While off-the-shelf billion-scale datasets for image generation are available, collecting similar video data of the same scale is still challenging. Also, training a video diffusion model is computationally much more expensive than its image counterpart. In this work, we explore finetuning a pretrained image diffusion model with video data as a practical solution for the video synthesis task. We find that naively extending the image noise prior to video noise prior in video diffusion leads to sub-optimal performance. Our carefully designed video noise prior leads to substantially better performance. Extensive experimental validation shows that our model, Preserve Your Own Correlation (PYoCo), attains SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks. It also achieves SOTA video generation quality on the small-scale UCF-101 benchmark with a $10\times$ smaller model using significantly less computation than the prior art.

翻译：尽管利用扩散模型生成高质量图像取得了巨大进展，但合成既逼真又时间一致性的动画帧序列仍处于起步阶段。虽然存在现成的数十亿规模图像生成数据集，但收集同等规模的视频数据仍具挑战。此外，训练视频扩散模型的计算成本远高于图像扩散模型。本研究探索了一种实用解决方案：使用视频数据微调预训练图像扩散模型以完成视频合成任务。我们发现，直接将图像噪声先验扩展到视频扩散中的视频噪声先验会导致次优性能。我们精心设计的视频噪声先验显著提升了性能。大量实验验证表明，我们的模型Preserve Your Own Correlation (PYoCo) 在UCF-101和MSR-VTT基准测试中实现了零样本文本生成视频的最新结果。同时，该模型在小型UCF-101基准测试中，以缩小10倍的模型规模和显著低于现有技术的计算量，达到了视频生成质量的最新水平。