We introduce Animate124 (Animate-one-image-to-4D), the first work to animate a single in-the-wild image into 3D video through textual motion descriptions, an underexplored problem with significant applications. Our 4D generation leverages an advanced 4D grid dynamic Neural Radiance Field (NeRF) model, optimized in three distinct stages using multiple diffusion priors. Initially, a static model is optimized using the reference image, guided by 2D and 3D diffusion priors, which serves as the initialization for the dynamic NeRF. Subsequently, a video diffusion model is employed to learn the motion specific to the subject. However, the object in the 3D videos tends to drift away from the reference image over time. This drift is mainly due to the misalignment between the text prompt and the reference image in the video diffusion model. In the final stage, a personalized diffusion prior is therefore utilized to address the semantic drift. As the pioneering image-text-to-4D generation framework, our method demonstrates significant advancements over existing baselines, evidenced by comprehensive quantitative and qualitative assessments.

翻译：我们提出Animate124（Animate-one-image-to-4D），这是首个通过文本运动描述将单张在野图像动画化为3D视频的工作，该问题此前研究不足但具有重要应用前景。我们的4D生成方法利用先进的4D网格动态神经辐射场（NeRF）模型，通过多阶段扩散先验进行三阶段优化。首先，基于参考图像，在2D和3D扩散先验指导下优化静态模型，作为动态NeRF的初始化。随后，采用视频扩散模型学习主体特有的运动模式。然而，3D视频中的物体会随时间推移偏离参考图像——这种漂移主要由视频扩散模型中文本提示与参考图像的语义错位导致。因此在最终阶段，引入个性化扩散先验以解决语义漂移问题。作为图像-文本到4D生成领域的开创性框架，我们的方法通过全面的定量与定性评估，展现出相较于现有基线的显著性能提升。