Large-scale diffusion generative models are greatly simplifying image, video and 3D asset creation from user-provided text prompts and images. However, the challenging problem of text-to-4D dynamic 3D scene generation with diffusion guidance remains largely unexplored. We propose Dream-in-4D, which features a novel two-stage approach for text-to-4D synthesis, leveraging (1) 3D and 2D diffusion guidance to effectively learn a high-quality static 3D asset in the first stage; (2) a deformable neural radiance field that explicitly disentangles the learned static asset from its deformation, preserving quality during motion learning; and (3) a multi-resolution feature grid for the deformation field with a displacement total variation loss to effectively learn motion with video diffusion guidance in the second stage. Through a user preference study, we demonstrate that our approach significantly advances image and motion quality, 3D consistency and text fidelity for text-to-4D generation compared to baseline approaches. Thanks to its motion-disentangled representation, Dream-in-4D can also be easily adapted for controllable generation where appearance is defined by one or multiple images, without the need to modify the motion learning stage. Thus, our method offers, for the first time, a unified approach for text-to-4D, image-to-4D and personalized 4D generation tasks.

翻译：大规模扩散生成模型极大地简化了基于用户文本提示和图像生成的图像、视频及三维资产创建流程。然而，利用扩散引导实现文本到四维动态三维场景生成这一具有挑战性的问题仍鲜有探索。我们提出Dream-in-4D方法，其创新性地采用两阶段文本到四维合成框架：(1)第一阶段结合三维与二维扩散引导，有效学习高质量静态三维资产；(2)采用可变形神经辐射场，将学习到的静态资产与其变形显式解耦，在运动学习过程中保持质量；(3)第二阶段对变形场采用多分辨率特征网格，结合位移总变分损失，在视频扩散引导下高效学习运动。通过用户偏好研究，我们证明该方法在文本到四维生成中，相较于基准方法，在图像与运动质量、三维一致性及文本保真度方面均取得显著提升。得益于运动解耦表示，Dream-in-4D可轻松适配外观由单幅或多幅图像定义的受控生成任务，无需修改运动学习阶段。因此，本方法首次为文本到四维、图像到四维及个性化四维生成任务提供了统一解决方案。