We present a novel differentiable point-based rendering framework for material and lighting decomposition from multi-view images, enabling editing, ray-tracing, and real-time relighting of the 3D point cloud. Specifically, a 3D scene is represented as a set of relightable 3D Gaussian points, where each point is additionally associated with a normal direction, BRDF parameters, and incident lights from different directions. To achieve robust lighting estimation, we further divide incident lights of each point into global and local components, as well as view-dependent visibilities. The 3D scene is optimized through the 3D Gaussian Splatting technique while BRDF and lighting are decomposed by physically-based differentiable rendering. Moreover, we introduce an innovative point-based ray-tracing approach based on the bounding volume hierarchy for efficient visibility baking, enabling real-time rendering and relighting of 3D Gaussian points with accurate shadow effects. Extensive experiments demonstrate improved BRDF estimation and novel view rendering results compared to state-of-the-art material estimation approaches. Our framework showcases the potential to revolutionize the mesh-based graphics pipeline with a relightable, traceable, and editable rendering pipeline solely based on point cloud. Project page:https://nju-3dv.github.io/projects/Relightable3DGaussian/.

翻译：我们提出一种新颖的可微分点基渲染框架，用于从多视角图像中分解材质与光照，实现3D点云的编辑、光线追踪与实时重光照。具体而言，3D场景被表示为一组可重光照的3D高斯点，每个点额外关联法线方向、BRDF参数以及来自不同方向的入射光。为实现稳健的光照估计，我们将每个点的入射光进一步分解为全局分量与局部分量，并考虑视角相关的可见性。该3D场景通过3D高斯溅射技术进行优化，同时借助基于物理的可微分渲染分解BRDF与光照。此外，我们引入一种基于层次包围盒的创新点基光线追踪方法，实现高效可见性烘焙，从而支持对3D高斯点进行带精确阴影效果的实时渲染与重光照。大量实验表明，与最先进的材质估计方法相比，我们提出的方法在BRDF估计与新视角渲染结果上均取得改进。该框架展示了仅基于点云即可颠覆传统基于网格图形管线的潜力，实现可重光照、可追踪且可编辑的渲染管线。项目页面：https://nju-3dv.github.io/projects/Relightable3DGaussian/