Diffusion models have achieved significant success in image and video generation. This motivates a growing interest in video editing tasks, where videos are edited according to provided text descriptions. However, most existing approaches only focus on video editing for short clips and rely on time-consuming tuning or inference. We are the first to propose Video Instruction Diffusion (VIDiff), a unified foundation model designed for a wide range of video tasks. These tasks encompass both understanding tasks (such as language-guided video object segmentation) and generative tasks (video editing and enhancement). Our model can edit and translate the desired results within seconds based on user instructions. Moreover, we design an iterative auto-regressive method to ensure consistency in editing and enhancing long videos. We provide convincing generative results for diverse input videos and written instructions, both qualitatively and quantitatively. More examples can be found at our website https://ChenHsing.github.io/VIDiff.

翻译：扩散模型在图像和视频生成中取得了显著成功，这激发了人们对视频编辑任务的日益增长的兴趣——即根据提供的文本描述对视频进行编辑。然而，现有方法大多仅关注短视频剪辑的编辑，且依赖耗时的调参或推理。我们首次提出视频指令扩散模型（VIDiff），这是一种专为广泛视频任务设计的统一基础模型，涵盖理解任务（如语言引导的视频目标分割）和生成任务（视频编辑与增强）。我们的模型可在数秒内根据用户指令编辑并翻译出期望结果。此外，我们设计了一种迭代自回归方法，以确保长视频编辑与增强的一致性。针对多样化的输入视频和书面指令，我们从定性和定量两方面提供了具有说服力的生成结果。更多示例请访问我们的网站 https://ChenHsing.github.io/VIDiff。