3D semantic scene understanding is a fundamental challenge in computer vision. It enables mobile agents to autonomously plan and navigate arbitrary environments. SSC formalizes this challenge as jointly estimating dense geometry and semantic information from sparse observations of a scene. Current methods for SSC are generally trained on 3D ground truth based on aggregated LiDAR scans. This process relies on special sensors and annotation by hand which are costly and do not scale well. To overcome this issue, our work presents the first self-supervised approach to SSC called S4C that does not rely on 3D ground truth data. Our proposed method can reconstruct a scene from a single image and only relies on videos and pseudo segmentation ground truth generated from off-the-shelf image segmentation network during training. Unlike existing methods, which use discrete voxel grids, we represent scenes as implicit semantic fields. This formulation allows querying any point within the camera frustum for occupancy and semantic class. Our architecture is trained through rendering-based self-supervised losses. Nonetheless, our method achieves performance close to fully supervised state-of-the-art methods. Additionally, our method demonstrates strong generalization capabilities and can synthesize accurate segmentation maps for far away viewpoints.

翻译：三维语义场景理解是计算机视觉中的一项基础挑战，它使移动智能体能够在任意环境中自主规划与导航。语义场景补全（SSC）将这一挑战形式化为：根据场景的稀疏观测数据联合估计稠密几何与语义信息。现有SSC方法通常基于聚合激光雷达扫描生成的三维真实数据进行训练。该过程依赖特殊传感器与人工标注，成本高昂且难以扩展。为解决这一问题，我们的工作提出了首个不依赖三维真实数据的自监督SSC方法——S4C。所提方法仅需单张图像即可重建场景，训练时仅依赖视频与现成图像分割网络生成的伪分割真值。与采用离散体素网格的现有方法不同，我们将场景表示为隐式语义场。这种公式化方法允许查询相机视锥内的任意点以获取占据状态与语义类别。我们的架构通过基于渲染的自监督损失进行训练。尽管如此，该方法在性能上仍接近全监督最先进方法。此外，本方法展现出强大的泛化能力，能够为远距离视点合成精确的分割图。