3D semantic scene understanding is a fundamental challenge in computer vision. It enables mobile agents to autonomously plan and navigate arbitrary environments. SSC formalizes this challenge as jointly estimating dense geometry and semantic information from sparse observations of a scene. Current methods for SSC are generally trained on 3D ground truth based on aggregated LiDAR scans. This process relies on special sensors and annotation by hand which are costly and do not scale well. To overcome this issue, our work presents the first self-supervised approach to SSC called S4C that does not rely on 3D ground truth data. Our proposed method can reconstruct a scene from a single image and only relies on videos and pseudo segmentation ground truth generated from off-the-shelf image segmentation network during training. Unlike existing methods, which use discrete voxel grids, we represent scenes as implicit semantic fields. This formulation allows querying any point within the camera frustum for occupancy and semantic class. Our architecture is trained through rendering-based self-supervised losses. Nonetheless, our method achieves performance close to fully supervised state-of-the-art methods. Additionally, our method demonstrates strong generalization capabilities and can synthesize accurate segmentation maps for far away viewpoints.

翻译：三维语义场景理解是计算机视觉中的一项基础性挑战，它使移动智能体能够自主规划和导航任意环境。语义场景补全（SSC）将这一挑战形式化为从场景的稀疏观测中联合估计稠密几何与语义信息。当前SSC方法通常基于聚合激光雷达扫描的三维真实标签进行训练。这一过程依赖特殊传感器和人工标注，成本高昂且扩展性差。为解决此问题，我们首次提出名为S4C的自监督语义场景补全方法，该方法无需三维真实标签数据。我们提出的方法能从单张图像重建场景，仅需依赖视频数据和现成图像分割网络生成的伪分割真实标签进行训练。与现有采用离散体素网格的方法不同，我们将场景表示为隐式语义场。这种表示形式允许查询相机视锥内的任意点的占用状态和语义类别。我们的架构通过基于渲染的自监督损失进行训练。尽管如此，我们的方法仍取得了接近全监督最优方法的性能。此外，该方法展现出强大的泛化能力，能为远距离视角合成精确的分割图。