3D semantic scene completion (SSC) is an ill-posed task that requires inferring a dense 3D scene from incomplete observations. Previous methods either explicitly incorporate 3D geometric input or rely on learnt 3D prior behind monocular RGB images. However, 3D sensors such as LiDAR are expensive and intrusive while monocular cameras face challenges in modeling precise geometry due to the inherent ambiguity. In this work, we propose StereoScene for 3D Semantic Scene Completion (SSC), which explores taking full advantage of light-weight camera inputs without resorting to any external 3D sensors. Our key insight is to leverage stereo matching to resolve geometric ambiguity. To improve its robustness in unmatched areas, we introduce bird's-eye-view (BEV) representation to inspire hallucination ability with rich context information. On top of the stereo and BEV representations, a mutual interactive aggregation (MIA) module is carefully devised to fully unleash their power. Specifically, a Bi-directional Interaction Transformer (BIT) augmented with confidence re-weighting is used to encourage reliable prediction through mutual guidance while a Dual Volume Aggregation (DVA) module is designed to facilitate complementary aggregation. Experimental results on SemanticKITTI demonstrate that the proposed StereoScene outperforms the state-of-the-art camera-based methods by a large margin with a relative improvement of 26.9% in geometry and 38.6% in semantic.

翻译：三维语义场景补全（SSC）是一项病态任务，要求从不完整观测中推断出稠密的三维场景。现有方法要么显式引入三维几何输入，要么依赖单目RGB图像背后的学习先验。然而，激光雷达等三维传感器成本高昂且具有侵入性，而单目相机因固有的模糊性难以精确建模几何结构。本文提出用于三维语义场景补全的立体场景（StereoScene）方法，旨在充分利用轻量级相机输入，无需依赖任何外部三维传感器。我们的核心洞察是利用立体匹配解决几何模糊性。为提升其在无匹配区域的鲁棒性，我们引入鸟瞰图（BEV）表示，以丰富的上下文信息激发幻觉能力。基于立体与BEV表示，我们精心设计了双向交互聚合（MIA）模块以充分释放其潜力。具体而言，采用置信度重加权增强的双向交互Transformer（BIT）通过相互引导促进可靠预测，同时设计双体素聚合（DVA）模块实现互补聚合。在SemanticKITTI数据集上的实验表明，所提出的立体场景方法大幅超越现有最优基于相机的方法，几何和语义分别相对提升26.9%和38.6%。