We present MOFA-Video, an advanced controllable image animation method that generates video from the given image using various additional controllable signals (such as human landmarks reference, manual trajectories, and another even provided video) or their combinations. This is different from previous methods which only can work on a specific motion domain or show weak control abilities with diffusion prior. To achieve our goal, we design several domain-aware motion field adapters (\ie, MOFA-Adapters) to control the generated motions in the video generation pipeline. For MOFA-Adapters, we consider the temporal motion consistency of the video and generate the dense motion flow from the given sparse control conditions first, and then, the multi-scale features of the given image are wrapped as a guided feature for stable video diffusion generation. We naively train two motion adapters for the manual trajectories and the human landmarks individually since they both contain sparse information about the control. After training, the MOFA-Adapters in different domains can also work together for more controllable video generation.

翻译：本文提出MOFA-Video，这是一种先进的可控图像动画方法，能够利用给定的图像结合各种附加可控信号（如人体关键点参考、手动轨迹，甚至另一段提供的视频）或其组合来生成视频。这与先前方法形成鲜明对比，先前方法仅能在特定运动域上工作，或在扩散先验下表现出较弱的控制能力。为实现这一目标，我们设计了多个域感知运动场适配器（即MOFA-Adapters）来控制在视频生成流程中的生成运动。对于MOFA-Adapters，我们考虑了视频的时间运动一致性，首先从给定的稀疏控制条件生成稠密运动流，然后将给定图像的多尺度特征进行形变包裹，作为稳定视频扩散生成的引导特征。由于手动轨迹和人体关键点均包含稀疏控制信息，我们分别对这两种运动适配器进行了独立训练。训练完成后，不同领域的MOFA-Adapters能够协同工作，实现更高可控性的视频生成。