The insertion of objects into a scene and relighting are commonly utilized applications in augmented reality (AR). Previous methods focused on inserting virtual objects using CAD models or real objects from single-view images, resulting in highly limited AR application scenarios. We propose a novel NeRF-based pipeline for inserting object NeRFs into scene NeRFs, enabling novel view synthesis and realistic relighting, supporting physical interactions like casting shadows onto each other, from two sets of images depicting the object and scene. The lighting environment is in a hybrid representation of Spherical Harmonics and Spherical Gaussians, representing both high- and low-frequency lighting components very well, and supporting non-Lambertian surfaces. Specifically, we leverage the benefits of volume rendering and introduce an innovative approach for efficient shadow rendering by comparing the depth maps between the camera view and the light source view and generating vivid soft shadows. The proposed method achieves realistic relighting effects in extensive experimental evaluations.

翻译：物体插入与重光照是增强现实（AR）中常用的应用场景。现有方法主要侧重于使用CAD模型或单视图图像中的真实物体进行虚拟物体插入，导致AR应用场景受到极大限制。本文提出一种新颖的基于神经辐射场（NeRF）的流程，可将物体NeRF插入场景NeRF中，仅需物体与场景的两组图像即可实现新视角合成与逼真的重光照效果，并支持物体间投射阴影等物理交互。本方法采用球谐函数与球面高斯函数的混合表示来建模光照环境，能够同时高质量表征高频与低频光照分量，并支持非朗伯表面。具体而言，我们充分利用体渲染的优势，通过比对相机视角与光源视角的深度图并生成生动的柔和阴影，提出了一种创新的高效阴影渲染方法。大量实验评估表明，所提方法能够实现逼真的重光照效果。