Reconstructing dynamic 3D scenes from 2D images and generating diverse views over time is challenging due to scene complexity and temporal dynamics. Despite advancements in neural implicit models, limitations persist: (i) Inadequate Scene Structure: Existing methods struggle to reveal the spatial and temporal structure of dynamic scenes from directly learning the complex 6D plenoptic function. (ii) Scaling Deformation Modeling: Explicitly modeling scene element deformation becomes impractical for complex dynamics. To address these issues, we consider the spacetime as an entirety and propose to approximate the underlying spatio-temporal 4D volume of a dynamic scene by optimizing a collection of 4D primitives, with explicit geometry and appearance modeling. Learning to optimize the 4D primitives enables us to synthesize novel views at any desired time with our tailored rendering routine. Our model is conceptually simple, consisting of a 4D Gaussian parameterized by anisotropic ellipses that can rotate arbitrarily in space and time, as well as view-dependent and time-evolved appearance represented by the coefficient of 4D spherindrical harmonics. This approach offers simplicity, flexibility for variable-length video and end-to-end training, and efficient real-time rendering, making it suitable for capturing complex dynamic scene motions. Experiments across various benchmarks, including monocular and multi-view scenarios, demonstrate our 4DGS model's superior visual quality and efficiency.

翻译：从二维图像重建动态三维场景并随时间生成多样化视角极具挑战性，这源于场景复杂性与时间动态特性。尽管神经隐式模型取得进展，仍存在以下局限：(i) 场景结构不完善：现有方法难以通过直接学习复杂的六维全光函数揭示动态场景的时空结构；(ii) 形变建模规模化困境：显式建模场景元素形变对复杂动态场景而言难以实现。针对上述问题，我们将时空视为整体，提出通过优化一组具有显式几何与外观建模的4D基元来逼近动态场景的底层时空4D体积。通过定制化的渲染方案学习优化4D基元，可在任意目标时刻合成新颖视角。本模型概念简洁，包含由各向异性椭圆参数化的4D高斯体（可在时空域自由旋转），以及由4D球谐系数表征的视角相关与时变外观。该方法具备简洁性、可变长度视频适应性、端到端训练能力及高效实时渲染特性，特别适用于捕获复杂动态场景运动。在包括单目与多视角在内的多基准测试中，实验表明我们的4DGS模型在视觉质量与效率方面均表现优异。