Text-driven video generation witnesses rapid progress. However, merely using text prompts is not enough to depict the desired subject appearance that accurately aligns with users' intents, especially for customized content creation. In this paper, we study the task of video generation with image prompts, which provide more accurate and direct content control beyond the text prompts. Specifically, we propose a feed-forward framework VideoBooth, with two dedicated designs: 1) We propose to embed image prompts in a coarse-to-fine manner. Coarse visual embeddings from image encoder provide high-level encodings of image prompts, while fine visual embeddings from the proposed attention injection module provide multi-scale and detailed encoding of image prompts. These two complementary embeddings can faithfully capture the desired appearance. 2) In the attention injection module at fine level, multi-scale image prompts are fed into different cross-frame attention layers as additional keys and values. This extra spatial information refines the details in the first frame and then it is propagated to the remaining frames, which maintains temporal consistency. Extensive experiments demonstrate that VideoBooth achieves state-of-the-art performance in generating customized high-quality videos with subjects specified in image prompts. Notably, VideoBooth is a generalizable framework where a single model works for a wide range of image prompts with feed-forward pass.

翻译：文本驱动的视频生成技术取得了快速发展。然而，仅使用文本提示不足以精确描绘与用户意图高度一致的目标主体外观，尤其是在定制化内容生成场景中。本文研究了基于图像提示的视频生成任务，该类提示能提供超越文本提示的更精准、更直接的内容控制。具体而言，我们提出了一种前馈式框架VideoBooth，包含两项专门设计：1) 我们提出以由粗到精的方式嵌入图像提示。图像编码器生成的粗粒度视觉嵌入提供图像提示的高层编码，而所提出的注意力注入模块生成的细粒度视觉嵌入则提供图像提示的多尺度与细节编码。这两种互补的嵌入能忠实捕捉所需的外观特征。2) 在细粒度的注意力注入模块中，多尺度图像提示作为额外的键值对输入到不同跨帧注意力层。这种额外的空间信息可优化首帧细节，并进一步传播至其余帧，从而维持时序一致性。大量实验表明，VideoBooth在生成由图像提示指定主体的定制化高质量视频方面达到了最先进性能。值得注意的是，VideoBooth是一个通用框架，单个模型通过前馈过程即可适用于各类图像提示。