Gaussian Splatting has recently become one of the most popular frameworks for photorealistic 3D scene reconstruction and rendering. While current rasterizers allow for efficient mappings of 3D Gaussian splats onto 2D camera views, this work focuses on mapping 2D image information (e.g. color, neural features or segmentation masks) efficiently back onto an existing scene of Gaussian splats. This 'opposite' direction enables applications ranging from scene relighting and stylization to 3D semantic segmentation, but also introduces challenges, such as view-dependent colorization and occlusion handling. Our approach tackles these challenges using the normal equation to solve a visibility-weighted least squares problem for every Gaussian and can be implemented efficiently with existing differentiable rasterizers. We demonstrate the effectiveness of our approach on scene relighting, feature enrichment and 3D semantic segmentation tasks, achieving up to an order of magnitude speedup compared to gradient descent-based baselines.

翻译：高斯溅印最近已成为最流行的框架之一，用于逼真的三维场景重建和渲染。虽然当前的光栅化器允许高效地将三维高斯溅印映射到二维相机视图，但本工作聚焦于反向操作：将二维图像信息（如颜色、神经特征或分割掩码）高效地映射回现有高斯溅印场景。这一“反向”方向支持从场景重照明和风格化到三维语义分割等多种应用，但也带来了诸如视角依赖色彩化和遮挡处理等挑战。我们的方法利用正规方程解决每个高斯点的可见性加权最小二乘问题，并可通过现有可微光栅化器高效实现。我们在场景重照明、特征增强和三维语义分割任务上展示了方法的有效性，相比基于梯度下降的基线方法，实现了高达一个数量级的加速。