In recent years, diffusion models have made remarkable strides in text-to-video generation, sparking a quest for enhanced control over video outputs to more accurately reflect user intentions. Traditional efforts predominantly focus on employing either semantic cues, like images or depth maps, or motion-based conditions, like moving sketches or object bounding boxes. Semantic inputs offer a rich scene context but lack detailed motion specificity; conversely, motion inputs provide precise trajectory information but miss the broader semantic narrative. For the first time, we integrate both semantic and motion cues within a diffusion model for video generation, as demonstrated in Fig 1. To this end, we introduce the Scene and Motion Conditional Diffusion (SMCD), a novel methodology for managing multimodal inputs. It incorporates a recognized motion conditioning module and investigates various approaches to integrate scene conditions, promoting synergy between different modalities. For model training, we separate the conditions for the two modalities, introducing a two-stage training pipeline. Experimental results demonstrate that our design significantly enhances video quality, motion precision, and semantic coherence.

翻译：近年来，扩散模型在文生视频领域取得了显著进展，促使研究者寻求对视频输出更精细的控制以准确反映用户意图。传统方法主要聚焦于利用语义线索（如图像或深度图）或运动条件（如动态草图或目标边界框）。语义输入能提供丰富的场景上下文，但缺乏详细的运动特异性；相反，运动输入能提供精确的轨迹信息，却缺失了更广泛的语义叙事。我们首次在扩散模型的视频生成中整合了语义与运动条件，如图1所示。为此，我们提出了场景与运动条件扩散模型（Scene and Motion Conditional Diffusion, SMCD），这是一种管理多模态输入的新方法。该模型融合了受认可的运动条件模块，并探索了多种整合场景条件的方法，以促进不同模态间的协同。在模型训练阶段，我们将两种模态的条件分离，引入两阶段训练流程。实验结果表明，我们的设计显著提升了视频质量、运动精度与语义连贯性。