3D semantic scene completion (SSC) is an ill-posed task that requires inferring a dense 3D scene from incomplete observations. Previous methods either explicitly incorporate 3D geometric input or rely on learnt 3D prior behind monocular RGB images. However, 3D sensors such as LiDAR are expensive and intrusive while monocular cameras face challenges in modeling precise geometry due to the inherent ambiguity. In this work, we propose StereoScene for 3D Semantic Scene Completion (SSC), which explores taking full advantage of light-weight camera inputs without resorting to any external 3D sensors. Our key insight is to leverage stereo matching to resolve geometric ambiguity. To improve its robustness in unmatched areas, we introduce bird's-eye-view (BEV) representation to inspire hallucination ability with rich context information. On top of the stereo and BEV representations, a mutual interactive aggregation (MIA) module is carefully devised to fully unleash their power. Specifically, a Bi-directional Interaction Transformer (BIT) augmented with confidence re-weighting is used to encourage reliable prediction through mutual guidance while a Dual Volume Aggregation (DVA) module is designed to facilitate complementary aggregation. Experimental results on SemanticKITTI demonstrate that the proposed StereoScene outperforms the state-of-the-art camera-based methods by a large margin with a relative improvement of 26.9% in geometry and 38.6% in semantic.

翻译：3D语义场景补全（SSC）是一项病态任务，要求从不完整观测中推断出稠密三维场景。现有方法要么显式引入3D几何输入，要么依赖单目RGB图像背后的学习先验。然而，激光雷达等3D传感器成本高昂且具侵入性，而单目相机因固有歧义性而在精确建模几何方面面临挑战。本文提出用于3D语义场景补全的StereoScene方法，探索充分利用轻量相机输入，无需依赖任何外部3D传感器。核心思路是利用立体匹配解决几何歧义问题。为提升其在非匹配区域的鲁棒性，我们引入鸟瞰图表示，通过丰富的上下文信息激发幻觉能力。在立体与BEV表示基础上，精心设计了互交互聚合模块以充分释放其潜力。具体而言，采用置信度重加权增强的双向交互Transformer，通过相互引导促进可靠预测；同时设计双体积聚合模块实现互补聚合。在SemanticKITTI数据集上的实验表明，所提StereoScene方法大幅超越现有基于相机的先进方法，几何和语义指标分别相对提升26.9%和38.6%。