We introduce InstructVid2Vid, an end-to-end diffusion-based methodology for video editing guided by human language instructions. Our approach empowers video manipulation guided by natural language directives, eliminating the need for per-example fine-tuning or inversion. The proposed InstructVid2Vid model modifies a pretrained image generation model, Stable Diffusion, to generate a time-dependent sequence of video frames. By harnessing the collective intelligence of disparate models, we engineer a training dataset rich in video-instruction triplets, which is a more cost-efficient alternative to collecting data in real-world scenarios. To enhance the coherence between successive frames within the generated videos, we propose the Inter-Frames Consistency Loss and incorporate it during the training process. With multimodal classifier-free guidance during the inference stage, the generated videos is able to resonate with both the input video and the accompanying instructions. Experimental results demonstrate that InstructVid2Vid is capable of generating high-quality, temporally coherent videos and performing diverse edits, including attribute editing, background changes, and style transfer. These results underscore the versatility and effectiveness of our proposed method.

翻译：我们提出了InstructVid2Vid，一种基于扩散模型的端到端方法，用于根据人类语言指令指导视频编辑。我们的方法实现了基于自然语言指令的视频操控，无需针对每个示例进行微调或反转。所提出的InstructVid2Vid模型通过修改预训练的图像生成模型Stable Diffusion，以生成时间相关的视频帧序列。通过利用不同模型的集体智慧，我们构建了一个富含视频-指令三元组的训练数据集，这比在真实场景中收集数据更具成本效益。为了增强生成视频中连续帧之间的一致性，我们提出了帧间一致性损失，并将其纳入训练过程。在推理阶段，通过采用多模态无分类器引导，生成的视频能够同时与输入视频和伴随指令产生共鸣。实验结果表明，InstructVid2Vid能够生成高质量、时间连贯的视频，并执行多样化的编辑，包括属性编辑、背景更改和风格迁移。这些结果突显了我们所提出方法的通用性和有效性。