The text-driven image and video diffusion models have achieved unprecedented success in generating realistic and diverse content. Recently, the editing and variation of existing images and videos in diffusion-based generative models have garnered significant attention. However, previous works are limited to editing content with text or providing coarse personalization using a single visual clue, rendering them unsuitable for indescribable content that requires fine-grained and detailed control. In this regard, we propose a generic video editing framework called Make-A-Protagonist, which utilizes textual and visual clues to edit videos with the goal of empowering individuals to become the protagonists. Specifically, we leverage multiple experts to parse source video, target visual and textual clues, and propose a visual-textual-based video generation model that employs mask-guided denoising sampling to generate the desired output. Extensive results demonstrate the versatile and remarkable editing capabilities of Make-A-Protagonist.

翻译：文本驱动的图像和视频扩散模型在生成逼真且多样化的内容方面取得了前所未有的成功。近年来，基于扩散生成模型的现有图像和视频编辑与变体技术引起了广泛关注。然而，先前的工作仅限于通过文本编辑内容，或利用单一视觉线索提供粗粒度的个性化定制，这使得它们无法处理需要细粒度且精确控制的不可描述内容。针对这一问题，我们提出了一种名为Make-A-Protagonist的通用视频编辑框架，该框架利用文本和视觉线索编辑视频，旨在赋予个体成为主角的能力。具体而言，我们借助多个专家模型解析源视频、目标视觉和文本线索，并提出了一种基于视觉-文本的视频生成模型，该模型采用掩码引导的去噪采样来生成期望的输出。大量实验结果表明，Make-A-Protagonist具有多样且卓越的编辑能力。