The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos. Webpage: https://diffusion-tokenflow.github.io/

翻译：生成式人工智能革命近期已扩展至视频领域。然而，当前最先进的视频模型在视觉质量和对生成内容的用户控制方面仍落后于图像模型。本文提出一个框架，利用文本到图像扩散模型的能力实现文本驱动的视频编辑任务。具体而言，给定源视频和目标文本提示，该方法能够生成符合目标文本的高质量视频，同时保留输入视频的空间布局与运动特征。该方法基于一项关键观察：通过强制扩散特征空间的一致性，可以获得编辑后视频的一致性。我们利用模型中已具备的帧间对应关系，显式传播扩散特征来实现这一目标。因此，本框架无需任何训练或微调，可与任何现成的文本到图像编辑方法配合使用。我们在多种真实世界视频上展示了最先进的编辑结果。网页地址：https://diffusion-tokenflow.github.io/