Reconstructing dynamic 3D scenes from 2D images and generating diverse views over time is challenging due to scene complexity and temporal dynamics. Despite advancements in neural implicit models, limitations persist: (i) Inadequate Scene Structure: Existing methods struggle to reveal the spatial and temporal structure of dynamic scenes from directly learning the complex 6D plenoptic function. (ii) Scaling Deformation Modeling: Explicitly modeling scene element deformation becomes impractical for complex dynamics. To address these issues, we consider the spacetime as an entirety and propose to approximate the underlying spatio-temporal 4D volume of a dynamic scene by optimizing a collection of 4D primitives, with explicit geometry and appearance modeling. Learning to optimize the 4D primitives enables us to synthesize novel views at any desired time with our tailored rendering routine. Our model is conceptually simple, consisting of a 4D Gaussian parameterized by anisotropic ellipses that can rotate arbitrarily in space and time, as well as view-dependent and time-evolved appearance represented by the coefficient of 4D spherindrical harmonics. This approach offers simplicity, flexibility for variable-length video and end-to-end training, and efficient real-time rendering, making it suitable for capturing complex dynamic scene motions. Experiments across various benchmarks, including monocular and multi-view scenarios, demonstrate our 4DGS model's superior visual quality and efficiency.

翻译：从二维图像重建动态三维场景并随时间生成多样化视图，因场景复杂度与时序动态性而面临挑战。尽管神经隐式模型取得进展，仍存在局限：(i) 场景结构不足：现有方法难以通过直接学习复杂的6维全光函数揭示动态场景的空时结构。(ii) 变形建模扩展性：显式建模场景元素变形对复杂动态场景而言变得不切实际。为解决这些问题，我们将时空视为整体，提出通过优化一组具有显式几何与外观建模的4维基元，逼近动态场景的潜在时空四维体。学习优化四维基元使我们能够通过定制渲染流程，在任意时刻合成新颖视角。本模型概念简洁，包含各向异性椭圆参数化的4维高斯，可在空间与时间中自由旋转，并通过4维球谐系数表达的视角依赖及时序演化外观。该方法简洁灵活，适用于变长视频与端到端训练，并支持高效实时渲染，从而胜任复杂动态场景运动的捕捉。在涵盖单目与多视角场景的多个基准测试中，实验表明我们的4DGS模型在视觉质量与效率上均具有优越性。