Reconstruction and intrinsic decomposition of scenes from captured imagery would enable many applications such as relighting and virtual object insertion. Recent NeRF based methods achieve impressive fidelity of 3D reconstruction, but bake the lighting and shadows into the radiance field, while mesh-based methods that facilitate intrinsic decomposition through differentiable rendering have not yet scaled to the complexity and scale of outdoor scenes. We present a novel inverse rendering framework for large urban scenes capable of jointly reconstructing the scene geometry, spatially-varying materials, and HDR lighting from a set of posed RGB images with optional depth. Specifically, we use a neural field to account for the primary rays, and use an explicit mesh (reconstructed from the underlying neural field) for modeling secondary rays that produce higher-order lighting effects such as cast shadows. By faithfully disentangling complex geometry and materials from lighting effects, our method enables photorealistic relighting with specular and shadow effects on several outdoor datasets. Moreover, it supports physics-based scene manipulations such as virtual object insertion with ray-traced shadow casting.

翻译：我们从拍摄的图像中重建场景并进行固有分解，能够实现重光照和虚拟物体插入等多种应用。近期基于NeRF的方法在三维重建保真度方面取得了令人印象深刻的成果，但将光照和阴影烘焙到辐射场中；而通过可微渲染实现固有分解的网格基方法尚未能扩展到户外场景的复杂性和规模。我们提出了一种面向大型城市场景的新型逆渲染框架，能够从一组带有可选深度的已知位姿RGB图像中联合重建场景几何、空间变化材质和高动态范围光照。具体而言，我们使用神经场处理主光线，并利用显式网格（由底层神经场重建）建模产生高阶光照效应（如投射阴影）的次级光线。通过将复杂几何和材质与光照效应进行忠实的解耦，我们的方法能够在多个户外数据集上实现具有镜面反射和阴影效果的照片级重光照。此外，它还支持基于物理的场景操作，例如通过光线追踪阴影投射进行虚拟物体插入。