Diffusion models have transformed the image-to-image (I2I) synthesis and are now permeating into videos. However, the advancement of video-to-video (V2V) synthesis has been hampered by the challenge of maintaining temporal consistency across video frames. This paper proposes a consistent V2V synthesis framework by jointly leveraging spatial conditions and temporal optical flow clues within the source video. Contrary to prior methods that strictly adhere to optical flow, our approach harnesses its benefits while handling the imperfection in flow estimation. We encode the optical flow via warping from the first frame and serve it as a supplementary reference in the diffusion model. This enables our model for video synthesis by editing the first frame with any prevalent I2I models and then propagating edits to successive frames. Our V2V model, FlowVid, demonstrates remarkable properties: (1) Flexibility: FlowVid works seamlessly with existing I2I models, facilitating various modifications, including stylization, object swaps, and local edits. (2) Efficiency: Generation of a 4-second video with 30 FPS and 512x512 resolution takes only 1.5 minutes, which is 3.1x, 7.2x, and 10.5x faster than CoDeF, Rerender, and TokenFlow, respectively. (3) High-quality: In user studies, our FlowVid is preferred 45.7% of the time, outperforming CoDeF (3.5%), Rerender (10.2%), and TokenFlow (40.4%).

翻译：扩散模型已彻底改变了图像到图像（I2I）合成，并正逐步渗透至视频领域。然而，视频到视频（V2V）合成的发展仍受制于跨帧时间一致性的挑战。本文提出一种一致的V2V合成框架，通过联合利用源视频中的空间条件和时序光流线索实现。与先前严格遵循光流的方法不同，我们的方法在驾驭其优势的同时处理光流估计中的不完善性。我们通过第一帧的扭曲操作编码光流，并将其作为补充参考输入扩散模型。这使得我们的模型能够通过先用任意流行的I2I模型编辑第一帧，再将编辑结果传播至后续帧的方式实现视频合成。我们的V2V模型FlowVid具有显著特性：（1）灵活性：FlowVid与现有I2I模型无缝协作，支持风格化、对象替换和局部编辑等多种修改。（2）高效率：生成30 FPS、512×512分辨率的4秒视频仅需1.5分钟，分别比CoDeF、Rerender和TokenFlow快3.1倍、7.2倍和10.5倍。（3）高质量：在用户研究中，FlowVid的偏好率达45.7%，优于CoDeF（3.5%）、Rerender（10.2%）和TokenFlow（40.4%）。