3D Semantic Scene Completion (SSC) can provide dense geometric and semantic scene representations, which can be applied in the field of autonomous driving and robotic systems. It is challenging to estimate the complete geometry and semantics of a scene solely from visual images, and accurate depth information is crucial for restoring 3D geometry. In this paper, we propose the first stereo SSC method named OccDepth, which fully exploits implicit depth information from stereo images (or RGBD images) to help the recovery of 3D geometric structures. The Stereo Soft Feature Assignment (Stereo-SFA) module is proposed to better fuse 3D depth-aware features by implicitly learning the correlation between stereo images. In particular, when the input are RGBD image, a virtual stereo images can be generated through original RGB image and depth map. Besides, the Occupancy Aware Depth (OAD) module is used to obtain geometry-aware 3D features by knowledge distillation using pre-trained depth models. In addition, a reformed TartanAir benchmark, named SemanticTartanAir, is provided in this paper for further testing our OccDepth method on SSC task. Compared with the state-of-the-art RGB-inferred SSC method, extensive experiments on SemanticKITTI show that our OccDepth method achieves superior performance with improving +4.82% mIoU, of which +2.49% mIoU comes from stereo images and +2.33% mIoU comes from our proposed depth-aware method. Our code and trained models are available at https://github.com/megvii-research/OccDepth.

翻译：3D语义场景补全（SSC）能够提供密集的几何与语义场景表示，可应用于自动驾驶和机器人系统领域。仅凭视觉图像估计场景的完整几何与语义信息具有挑战性，而准确的深度信息对恢复3D几何至关重要。本文提出首个立体SSC方法——OccDepth，该方法充分利用立体图像（或RGBD图像）中的隐式深度信息来辅助3D几何结构的恢复。我们提出立体软特征分配（Stereo-SFA）模块，通过隐式学习立体图像间的相关性，更好地融合3D深度感知特征。特别地，当输入为RGBD图像时，可通过原始RGB图像与深度图生成虚拟立体图像。此外，占用感知深度（OAD）模块利用预训练深度模型进行知识蒸馏，获取几何感知的3D特征。本文还提供了重构后的TartanAir基准——SemanticTartanAir，用于进一步测试OccDepth方法在SSC任务上的性能。与最先进的RGB推断SSC方法相比，在SemanticKITTI上的大量实验表明，OccDepth方法实现了优越性能，mIoU提升+4.82%，其中+2.49%的mIoU来自立体图像，+2.33%的mIoU来自我们提出的深度感知方法。我们的代码和训练模型已开源在https://github.com/megvii-research/OccDepth。