Novel view synthesis of dynamic scenes has been an intriguing yet challenging problem. Despite recent advancements, simultaneously achieving high-resolution photorealistic results, real-time rendering, and compact storage remains a formidable task. To address these challenges, we propose Spacetime Gaussian Feature Splatting as a novel dynamic scene representation, composed of three pivotal components. First, we formulate expressive Spacetime Gaussians by enhancing 3D Gaussians with temporal opacity and parametric motion/rotation. This enables Spacetime Gaussians to capture static, dynamic, as well as transient content within a scene. Second, we introduce splatted feature rendering, which replaces spherical harmonics with neural features. These features facilitate the modeling of view- and time-dependent appearance while maintaining small size. Third, we leverage the guidance of training error and coarse depth to sample new Gaussians in areas that are challenging to converge with existing pipelines. Experiments on several established real-world datasets demonstrate that our method achieves state-of-the-art rendering quality and speed, while retaining compact storage. At 8K resolution, our lite-version model can render at 60 FPS on an Nvidia RTX 4090 GPU. Our code is available at https://github.com/oppo-us-research/SpacetimeGaussians.

翻译：动态场景的新视角合成一直是一个引人入胜但极具挑战的问题。尽管近年来取得了进展，但同步实现高分辨率逼真渲染、实时渲染与紧凑存储仍然是一项艰巨任务。为应对这些挑战，我们提出时空高斯特征喷溅作为一种新型动态场景表示方法，其包含三个关键组成部分。首先，我们通过为3D高斯引入时间不透明度与参数化运动/旋转，构建了具有高表达力的时空高斯单元。这使得时空高斯能够捕捉场景中的静态、动态及瞬态内容。其次，我们引入喷溅特征渲染，用神经特征取代球谐函数。这些特征在保持小尺寸的同时，能够有效建模视角相关与时间相关的外观。第三，我们利用训练误差与粗略深度的引导，在现有流程难以收敛的区域采样新的高斯单元。在多个已建立的真实世界数据集上的实验表明，我们的方法在保持紧凑存储的同时，实现了最先进的渲染质量与速度。在8K分辨率下，我们的精简版模型能在Nvidia RTX 4090 GPU上以60 FPS的速度进行渲染。我们的代码已开源，详见https://github.com/oppo-us-research/SpacetimeGaussians。