Immersive novel view generation is an important technology in the field of graphics and has recently also received attention for operator-based human-robot interaction. However, the involved training is time-consuming, and thus the current test scope is majorly on object capturing. This limits the usage of related models in the robotics community for 3D reconstruction since robots (1) usually only capture a very small range of view directions to surfaces that cause arbitrary predictions on unseen, novel direction, (2) requires real-time algorithms, and (3) work with growing scenes, e.g., in robotic exploration. The paper proposes a novel Neural Surface Light Fields model that copes with the small range of view directions while producing a good result in unseen directions. Exploiting recent encoding techniques, the training of our model is highly efficient. In addition, we design Multiple Asynchronous Neural Agents (MANA), a universal framework to learn each small region in parallel for large-scale growing scenes. Our model learns online the Neural Surface Light Fields (NSLF) aside from real-time 3D reconstruction with a sequential data stream as the shared input. In addition to online training, our model also provides real-time rendering after completing the data stream for visualization. We implement experiments using well-known RGBD indoor datasets, showing the high flexibility to embed our model into real-time 3D reconstruction and demonstrating high-fidelity view synthesis for these scenes. The code is available on github.

翻译：沉浸式新颖视角生成是图形学领域中的一项重要技术，近年来也受到基于操作者的人机交互领域的关注。然而，相关的训练过程耗时较长，因此目前的测试范围主要限于物体捕捉。这限制了相关模型在机器人社区的3D重建应用，因为机器人（1）通常只能捕捉到表面非常有限的视角方向，导致对未见、新颖方向的预测具有任意性；（2）需要实时算法；（3）处理不断增长的场景，例如在机器人探索中。本文提出了一种新颖的神经表面光场模型，该模型能够应对有限的视角方向范围，同时在新颖方向上产生良好的结果。利用最新的编码技术，我们模型的训练效率非常高。此外，我们设计了多异步神经代理（MANA）——一个通用框架，用于并行学习大规模增长场景中的每个小区域。我们的模型以共享输入的连续数据流为基础，在实时3D重建的同时在线学习神经表面光场（NSLF）。除了在线训练，我们的模型还在数据流完成后提供实时渲染以进行可视化。我们使用著名的RGBD室内数据集进行实验，展示了将我们的模型嵌入实时3D重建的高度灵活性，并证明了这些场景中的高保真度视图合成。代码可在GitHub上获取。