This paper studies the human image animation task, which aims to generate a video of a certain reference identity following a particular motion sequence. Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion. Despite achieving reasonable results, these approaches face challenges in maintaining temporal consistency throughout the animation due to the lack of temporal modeling and poor preservation of reference identity. In this work, we introduce MagicAnimate, a diffusion-based framework that aims at enhancing temporal consistency, preserving reference image faithfully, and improving animation fidelity. To achieve this, we first develop a video diffusion model to encode temporal information. Second, to maintain the appearance coherence across frames, we introduce a novel appearance encoder to retain the intricate details of the reference image. Leveraging these two innovations, we further employ a simple video fusion technique to encourage smooth transitions for long video animation. Empirical results demonstrate the superiority of our method over baseline approaches on two benchmarks. Notably, our approach outperforms the strongest baseline by over 38% in terms of video fidelity on the challenging TikTok dancing dataset. Code and model will be made available.

翻译：本文研究人体图像动画任务，旨在根据特定动作序列生成与参考身份一致的视频。现有动画工作通常采用帧扭曲技术将参考图像向目标动作驱动。尽管取得了一定成效，但由于缺乏时间建模且对参考身份的保持能力不足，这些方法在维持动画全程时间一致性方面面临挑战。本研究提出MagicAnimate——一个基于扩散的框架，旨在增强时间一致性、忠实保持参考图像并提升动画保真度。为此，我们首先开发视频扩散模型以编码时间信息；其次，为维持帧间外观连贯性，创新性地引入外观编码器保留参考图像的精细细节。基于这两项创新，我们进一步采用简易视频融合技术促进长视频动画的平滑过渡。实验结果表明，本方法在两个基准测试中均显著优于基线方法。值得注意的是，在具有挑战性的TikTok舞蹈数据集上，我们的方法在视频保真度方面超越最强基线方法超过38%。代码与模型将开源。