Recently, diffusion models like StableDiffusion have achieved impressive image generation results. However, the generation process of such diffusion models is uncontrollable, which makes it hard to generate videos with continuous and consistent content. In this work, by using the diffusion model with ControlNet, we proposed a new motion-guided video-to-video translation framework called VideoControlNet to generate various videos based on the given prompts and the condition from the input video. Inspired by the video codecs that use motion information for reducing temporal redundancy, our framework uses motion information to prevent the regeneration of the redundant areas for content consistency. Specifically, we generate the first frame (i.e., the I-frame) by using the diffusion model with ControlNet. Then we generate other key frames (i.e., the P-frame) based on the previous I/P-frame by using our newly proposed motion-guided P-frame generation (MgPG) method, in which the P-frames are generated based on the motion information and the occlusion areas are inpainted by using the diffusion model. Finally, the rest frames (i.e., the B-frame) are generated by using our motion-guided B-frame interpolation (MgBI) module. Our experiments demonstrate that our proposed VideoControlNet inherits the generation capability of the pre-trained large diffusion model and extends the image diffusion model to the video diffusion model by using motion information. More results are provided at our project page.

翻译：近期，StableDiffusion等扩散模型在图像生成领域取得了显著成果。然而，这类扩散模型的生成过程缺乏可控性，导致难以生成内容连续且一致的视频。本研究通过结合ControlNet与扩散模型，提出了一种名为VideoControlNet的新的运动引导视频到视频翻译框架，该框架能根据给定提示及输入视频的条件生成多样化视频。受视频编解码器利用运动信息减少时间冗余的启发，本框架采用运动信息避免冗余区域的重建，从而确保内容一致性。具体而言，我们首先利用结合ControlNet的扩散模型生成第一帧（即I帧）。随后，基于前一I/P帧，通过新提出的运动引导P帧生成（MgPG）方法生成其他关键帧（即P帧），其中P帧依据运动信息生成，而遮挡区域则通过扩散模型进行修补。最后，利用运动引导B帧插值（MgBI）模块生成剩余帧（即B帧）。实验证明，我们提出的VideoControlNet继承了预训练大型扩散模型的生成能力，并通过运动信息将图像扩散模型扩展至视频扩散模型。更多结果详见项目主页。