Current 4D generation methods have achieved noteworthy efficacy with the aid of advanced diffusion generative models. However, these methods lack multi-view spatial-temporal modeling and encounter challenges in integrating diverse prior knowledge from multiple diffusion models, resulting in inconsistent temporal appearance and flickers. In this paper, we propose a novel 4D generation pipeline, namely 4Diffusion aimed at generating spatial-temporally consistent 4D content from a monocular video. We first design a unified diffusion model tailored for multi-view video generation by incorporating a learnable motion module into a frozen 3D-aware diffusion model to capture multi-view spatial-temporal correlations. After training on a curated dataset, our diffusion model acquires reasonable temporal consistency and inherently preserves the generalizability and spatial consistency of the 3D-aware diffusion model. Subsequently, we propose 4D-aware Score Distillation Sampling loss, which is based on our multi-view video diffusion model, to optimize 4D representation parameterized by dynamic NeRF. This aims to eliminate discrepancies arising from multiple diffusion models, allowing for generating spatial-temporally consistent 4D content. Moreover, we devise an anchor loss to enhance the appearance details and facilitate the learning of dynamic NeRF. Extensive qualitative and quantitative experiments demonstrate that our method achieves superior performance compared to previous methods.

翻译：当前的四维生成方法借助先进的扩散生成模型已取得显著成效。然而，这些方法缺乏多视角时空建模，且在整合来自多个扩散模型的不同先验知识时面临挑战，导致时间外观不一致与闪烁现象。本文提出一种新颖的四维生成流程——4Diffusion，旨在从单目视频生成时空一致的四维内容。我们首先设计了一个专用于多视角视频生成的统一扩散模型，其方法是在冻结的三维感知扩散模型中嵌入可学习的运动模块，以捕捉多视角时空关联。在精选数据集上训练后，我们的扩散模型获得了合理的时间一致性，并内在保持了三维感知扩散模型的泛化能力与空间一致性。随后，我们提出基于多视角视频扩散模型的四维感知分数蒸馏采样损失，用以优化由动态神经辐射场参数化的四维表征。该设计旨在消除由多个扩散模型引起的差异，从而生成时空一致的四维内容。此外，我们设计了锚点损失以增强外观细节并促进动态神经辐射场的学习。大量定性与定量实验表明，相较于现有方法，我们的方法实现了更优越的性能。