Large text-to-image diffusion models have exhibited impressive proficiency in generating high-quality images. However, when applying these models to video domain, ensuring temporal consistency across video frames remains a formidable challenge. This paper proposes a novel zero-shot text-guided video-to-video translation framework to adapt image models to videos. The framework includes two parts: key frame translation and full video translation. The first part uses an adapted diffusion model to generate key frames, with hierarchical cross-frame constraints applied to enforce coherence in shapes, textures and colors. The second part propagates the key frames to other frames with temporal-aware patch matching and frame blending. Our framework achieves global style and local texture temporal consistency at a low cost (without re-training or optimization). The adaptation is compatible with existing image diffusion techniques, allowing our framework to take advantage of them, such as customizing a specific subject with LoRA, and introducing extra spatial guidance with ControlNet. Extensive experimental results demonstrate the effectiveness of our proposed framework over existing methods in rendering high-quality and temporally-coherent videos.

翻译：大型文本到图像扩散模型在生成高质量图像方面展现了卓越的能力。然而，将这些模型应用于视频领域时，确保帧间的时间一致性仍是一个严峻的挑战。本文提出了一种新颖的零样本文本引导视频到视频翻译框架，用于将图像模型适配到视频领域。该框架包含两个部分：关键帧翻译和全视频翻译。第一部分使用适配的扩散模型生成关键帧，并通过层次化跨帧约束强制保持形状、纹理和颜色的一致性。第二部分通过时间感知的块匹配和帧融合将关键帧传播到其他帧。我们的框架以低成本（无需重新训练或优化）实现了全局风格和局部纹理的时间一致性。该适配与现有图像扩散技术兼容，使我们的框架能够利用这些技术，例如通过LoRA定制特定主体，以及通过ControlNet引入额外的空间引导。大量实验结果表明，在生成高质量且时间一致的视频方面，我们提出的框架优于现有方法。