Recent video generative models primarily rely on carefully written text prompts for specific tasks, like inpainting or style editing. They require labor-intensive textual descriptions for input videos, hindering their flexibility to adapt personal/raw videos to user specifications. This paper proposes RACCooN, a versatile and user-friendly video-to-paragraph-to-video generative framework that supports multiple video editing capabilities such as removal, addition, and modification, through a unified pipeline. RACCooN consists of two principal stages: Video-to-Paragraph (V2P) and Paragraph-to-Video (P2V). In the V2P stage, we automatically describe video scenes in well-structured natural language, capturing both the holistic context and focused object details. Subsequently, in the P2V stage, users can optionally refine these descriptions to guide the video diffusion model, enabling various modifications to the input video, such as removing, changing subjects, and/or adding new objects. The proposed approach stands out from other methods through several significant contributions: (1) RACCooN suggests a multi-granular spatiotemporal pooling strategy to generate well-structured video descriptions, capturing both the broad context and object details without requiring complex human annotations, simplifying precise video content editing based on text for users. (2) Our video generative model incorporates auto-generated narratives or instructions to enhance the quality and accuracy of the generated content. It supports the addition of video objects, inpainting, and attribute modification within a unified framework, surpassing existing video editing and inpainting benchmarks. The proposed framework demonstrates impressive versatile capabilities in video-to-paragraph generation, video content editing, and can be incorporated into other SoTA video generative models for further enhancement.

翻译：近期视频生成模型主要依赖精心编写的文本提示来完成特定任务，如修复或风格编辑。这些方法需要对输入视频进行费时费力的文本描述，限制了其将个人/原始视频灵活适应用户需求的能力。本文提出RACCooN，一个通用且用户友好的视频-段落-视频生成框架，通过统一流程支持多种视频编辑功能，如移除、添加和修改。RACCooN包含两个主要阶段：视频到段落（V2P）和段落到视频（P2V）。在V2P阶段，我们以结构化的自然语言自动描述视频场景，同时捕捉整体上下文和聚焦的对象细节。随后在P2V阶段，用户可选择性地优化这些描述以指导视频扩散模型，从而实现对输入视频的多种修改，例如移除、改变主体和/或添加新对象。所提方法通过以下重要贡献区别于其他方法：（1）RACCooN提出多粒度时空池化策略来生成结构良好的视频描述，无需复杂人工标注即可同时捕捉宏观上下文和对象细节，简化了用户基于文本的精确视频内容编辑。（2）我们的视频生成模型融合自动生成的叙事或指令以提升生成内容的质量与准确性。该模型在统一框架内支持视频对象添加、修复和属性修改，性能超越现有视频编辑与修复基准。所提框架在视频-段落生成、视频内容编辑方面展现出卓越的通用能力，并可集成至其他先进视频生成模型以实现进一步性能提升。