We present an end-to-end diffusion-based method for editing videos with human language instructions, namely $\textbf{InstructVid2Vid}$. Our approach enables the editing of input videos based on natural language instructions without any per-example fine-tuning or inversion. The proposed InstructVid2Vid model combines a pretrained image generation model, Stable Diffusion, with a conditional 3D U-Net architecture to generate time-dependent sequence of video frames. To obtain the training data, we incorporate the knowledge and expertise of different models, including ChatGPT, BLIP, and Tune-a-Video, to synthesize video-instruction triplets, which is a more cost-efficient alternative to collecting data in real-world scenarios. To improve the consistency between adjacent frames of generated videos, we propose the Frame Difference Loss, which is incorporated during the training process. During inference, we extend the classifier-free guidance to text-video input to guide the generated results, making them more related to both the input video and instruction. Experiments demonstrate that InstructVid2Vid is able to generate high-quality, temporally coherent videos and perform diverse edits, including attribute editing, change of background, and style transfer. These results highlight the versatility and effectiveness of our proposed method. Code is released in $\href{https://github.com/BrightQin/InstructVid2Vid}{InstructVid2Vid}$.

翻译：我们提出了一种端到端的基于扩散模型的视频编辑方法，即 $\textbf{InstructVid2Vid}$，该方法能够通过人类语言指令对视频进行编辑。我们的方法无需针对每个样本进行微调或反转，即可根据自然语言指令直接编辑输入视频。所提出的InstructVid2Vid模型结合了预训练图像生成模型Stable Diffusion与条件3D U-Net架构，用于生成时间相关序列的视频帧。为获取训练数据，我们整合了ChatGPT、BLIP和Tune-a-Video等多种模型的知识与专长，合成了视频-指令三元组，相比在真实场景中收集数据更具成本效益。为提升生成视频相邻帧间的连贯性，我们提出了帧差损失，并将其融入训练过程。在推理阶段，我们将无分类器引导扩展至文本-视频输入，以指导生成结果，使其与输入视频和指令均更相关。实验表明，InstructVid2Vid能够生成高质量、时间连贯的视频，并执行多样化的编辑操作，包括属性编辑、背景更换及风格迁移。这些结果凸显了所提方法的通用性与有效性。代码已发布于 $\href{https://github.com/BrightQin/InstructVid2Vid}{InstructVid2Vid}$。