While open-source video generation and editing models have made significant progress, individual models are typically limited to specific tasks, failing to meet the diverse needs of users. Effectively coordinating these models can unlock a wide range of video generation and editing capabilities. However, manual coordination is complex and time-consuming, requiring users to deeply understand task requirements and possess comprehensive knowledge of each model's performance, applicability, and limitations, thereby increasing the barrier to entry. To address these challenges, we propose a novel video generation and editing system powered by our Semantic Planning Agent (SPAgent). SPAgent bridges the gap between diverse user intents and the effective utilization of existing generative models, enhancing the adaptability, efficiency, and overall quality of video generation and editing. Specifically, the SPAgent assembles a tool library integrating state-of-the-art open-source image and video generation and editing models as tools. After fine-tuning on our manually annotated dataset, SPAgent can automatically coordinate the tools for video generation and editing, through our novelly designed three-step framework: (1) decoupled intent recognition, (2) principle-guided route planning, and (3) capability-based execution model selection. Additionally, we enhance the SPAgent's video quality evaluation capability, enabling it to autonomously assess and incorporate new video generation and editing models into its tool library without human intervention. Experimental results demonstrate that the SPAgent effectively coordinates models to generate or edit videos, highlighting its versatility and adaptability across various video tasks.

翻译：尽管开源视频生成与编辑模型已取得显著进展，但单个模型通常局限于特定任务，难以满足用户的多样化需求。有效协调这些模型能够释放广泛的视频生成与编辑能力。然而，人工协调过程复杂且耗时，要求用户深入理解任务需求，并全面掌握每个模型的性能、适用场景与局限性，从而提高了使用门槛。为应对这些挑战，我们提出了一种由语义规划智能体（SPAgent）驱动的新型视频生成与编辑系统。SPAgent在多样化用户意图与现有生成模型的有效利用之间架起桥梁，提升了视频生成与编辑的适应性、效率与整体质量。具体而言，SPAgent构建了一个集成前沿开源图像及视频生成与编辑模型的工具库。在我们人工标注的数据集上进行微调后，SPAgent能够通过我们全新设计的三步框架自动协调工具以完成视频生成与编辑任务：（1）解耦意图识别，（2）原则引导的路径规划，以及（3）基于能力的执行模型选择。此外，我们增强了SPAgent的视频质量评估能力，使其能够自主评估新的视频生成与编辑模型，并在无需人工干预的情况下将其纳入工具库。实验结果表明，SPAgent能有效协调多个模型以生成或编辑视频，突显了其在各类视频任务中的多功能性与适应性。