Recent advancements in diffusion models have greatly improved the quality and diversity of synthesized content. To harness the expressive power of diffusion models, researchers have explored various controllable mechanisms that allow users to intuitively guide the content synthesis process. Although the latest efforts have primarily focused on video synthesis, there has been a lack of effective methods for controlling and describing desired content and motion. In response to this gap, we introduce MCDiff, a conditional diffusion model that generates a video from a starting image frame and a set of strokes, which allow users to specify the intended content and dynamics for synthesis. To tackle the ambiguity of sparse motion inputs and achieve better synthesis quality, MCDiff first utilizes a flow completion model to predict the dense video motion based on the semantic understanding of the video frame and the sparse motion control. Then, the diffusion model synthesizes high-quality future frames to form the output video. We qualitatively and quantitatively show that MCDiff achieves the state-the-of-art visual quality in stroke-guided controllable video synthesis. Additional experiments on MPII Human Pose further exhibit the capability of our model on diverse content and motion synthesis.

翻译：近期扩散模型的进展极大地提升了合成内容的质量与多样性。为利用扩散模型的强大表达能力，研究者探索了多种可控机制，使用户能够直观地引导内容合成过程。尽管最新工作主要聚焦于视频合成，但在有效描述和控制所需内容及运动的方法上仍存在不足。针对这一空白，我们提出MCDiff，一种条件扩散模型，可从起始图像帧和一组笔画生成视频，用户可通过这些笔画指定合成的内容与动态。为处理稀疏运动输入的歧义性并提升合成质量，MCDiff首先利用流完成模型基于视频帧的语义理解与稀疏运动控制预测密集视频运动，随后扩散模型合成高质量的未来帧以构成输出视频。我们定性与定量地证明，MCDiff在笔画引导的可控视频合成中达到了最先进的视觉质量。在MPII人体姿态上的额外实验进一步展示了模型在多内容与运动合成方面的能力。