Occlusion is a common issue in 3D reconstruction from RGB-D videos, often blocking the complete reconstruction of objects and presenting an ongoing problem. In this paper, we propose a novel framework, empowered by a 2D diffusion-based in-painting model, to reconstruct complete surfaces for the hidden parts of objects. Specifically, we utilize a pre-trained diffusion model to fill in the hidden areas of 2D images. Then we use these in-painted images to optimize a neural implicit surface representation for each instance for 3D reconstruction. Since creating the in-painting masks needed for this process is tricky, we adopt a human-in-the-loop strategy that involves very little human engagement to generate high-quality masks. Moreover, some parts of objects can be totally hidden because the videos are usually shot from limited perspectives. To ensure recovering these invisible areas, we develop a cascaded network architecture for predicting signed distance field, making use of different frequency bands of positional encoding and maintaining overall smoothness. Besides the commonly used rendering loss, Eikonal loss, and silhouette loss, we adopt a CLIP-based semantic consistency loss to guide the surface from unseen camera angles. Experiments on ScanNet scenes show that our proposed framework achieves state-of-the-art accuracy and completeness in object-level reconstruction from scene-level RGB-D videos.

翻译：遮挡是RGB-D视频三维重建中的常见问题，常阻碍物体的完整重建，至今仍是持续挑战。本文提出一种新型框架，借助基于2D扩散的修复模型，为物体的被遮挡部分重建完整表面。具体而言，我们利用预训练扩散模型填充二维图像的隐藏区域，随后使用修复后的图像优化每个实例的神经隐式表面表征，实现三维重建。由于生成该流程所需的修复掩膜较为困难，我们采用一种仅需极少人工参与的人机协同策略来生成高质量掩膜。此外，由于视频通常仅从有限视角拍摄，物体的某些部分可能完全不可见。为确保恢复这些不可见区域，我们开发了一种级联网络架构用于预测符号距离场，该架构利用位置编码的不同频段并保持整体平滑性。除常用的渲染损失、Eikonal损失和轮廓损失外，我们采用基于CLIP的语义一致性损失来引导来自未见过相机角度的表面重建。在ScanNet场景上的实验表明，本框架在基于场景级RGB-D视频的物体级重建中达到了最先进的精度与完整性。