Existing real-time RGB-D reconstruction approaches, like Kinect Fusion, lack real-time photo-realistic visualization. This is due to noisy, oversmoothed or incomplete geometry and blurry textures which are fused from imperfect depth maps and camera poses. Recent neural rendering methods can overcome many of such artifacts but are mostly optimized for offline usage, hindering the integration into a live reconstruction pipeline. In this paper, we present LiveNVS, a system that allows for neural novel view synthesis on a live RGB-D input stream with very low latency and real-time rendering. Based on the RGB-D input stream, novel views are rendered by projecting neural features into the target view via a densely fused depth map and aggregating the features in image-space to a target feature map. A generalizable neural network then translates the target feature map into a high-quality RGB image. LiveNVS achieves state-of-the-art neural rendering quality of unknown scenes during capturing, allowing users to virtually explore the scene and assess reconstruction quality in real-time.

翻译：现有的实时RGB-D重建方法（如Kinect Fusion）缺乏实时的逼真可视化能力。这是由于不完美的深度图和相机位姿融合导致的噪声、过度平滑或不完整的几何结构以及模糊纹理。近年来的神经渲染方法能够克服许多此类伪影，但大多针对离线使用优化，阻碍了其集成到实时重建管线中。本文提出LiveNVS系统，该支持在实时RGB-D输入流上以极低延迟和实时渲染进行神经新视角合成。基于RGB-D输入流，通过密集融合的深度图将神经特征投影到目标视角，并在图像空间中将特征聚合为目标特征图，从而渲染出新视角。随后，一个可泛化的神经网络将目标特征图转换为高质量的RGB图像。LiveNVS在采集过程中实现了未知场景的最先进神经渲染质量，使用户能够实时虚拟探索场景并评估重建质量。