Recent neural implicit representations (NIRs) have achieved great success in the tasks of 3D reconstruction and novel view synthesis. However, they require the images of a scene from different camera views to be available for one-time training. This is expensive especially for scenarios with large-scale scenes and limited data storage. In view of this, we explore the task of incremental learning for NIRs in this work. We design a student-teacher framework to mitigate the catastrophic forgetting problem. Specifically, we iterate the process of using the student as the teacher at the end of each time step and let the teacher guide the training of the student in the next step. As a result, the student network is able to learn new information from the streaming data and retain old knowledge from the teacher network simultaneously. Although intuitive, naively applying the student-teacher pipeline does not work well in our task. Not all information from the teacher network is helpful since it is only trained with the old data. To alleviate this problem, we further introduce a random inquirer and an uncertainty-based filter to filter useful information. Our proposed method is general and thus can be adapted to different implicit representations such as neural radiance field (NeRF) and neural surface field. Extensive experimental results for both 3D reconstruction and novel view synthesis demonstrate the effectiveness of our approach compared to different baselines.

翻译：近年来，神经隐式表示（NIRs）在三维重建和新视角合成任务中取得了巨大成功。然而，它们要求一次性训练时能获取场景来自不同相机视角的图像。这对于大规模场景和数据存储受限的场景而言成本高昂。鉴于此，本文探索了神经隐式表示的增量学习任务。我们设计了一个师生框架以缓解灾难性遗忘问题。具体而言，我们在每个时间步结束时将学生模型迭代作为教师，并让该教师在下一步指导学生模型的训练。因此，学生网络能够同时从流式数据中学习新信息，并从教师网络中保留旧知识。尽管直观，但在我们的任务中直接应用师生框架效果不佳。由于教师网络仅使用旧数据训练，其并非所有信息都有益。为缓解此问题，我们进一步引入了随机查询器和一个基于不确定性的过滤器来筛选有用信息。所提方法具有通用性，因此可适配于不同的隐式表示，如神经辐射场（NeRF）和神经表面场。在三维重建和新视角合成任务上的大量实验结果证明了本方法相较于不同基线的有效性。