Stereo matching and semantic segmentation are significant tasks in binocular satellite 3D reconstruction. However, previous studies primarily view these as independent parallel tasks, lacking an integrated multitask learning framework. This work introduces a solution, the Single-branch Semantic Stereo Network (S3Net), which innovatively combines semantic segmentation and stereo matching using Self-Fuse and Mutual-Fuse modules. Unlike preceding methods that utilize semantic or disparity information independently, our method dentifies and leverages the intrinsic link between these two tasks, leading to a more accurate understanding of semantic information and disparity estimation. Comparative testing on the US3D dataset proves the effectiveness of our S3Net. Our model improves the mIoU in semantic segmentation from 61.38 to 67.39, and reduces the D1-Error and average endpoint error (EPE) in disparity estimation from 10.051 to 9.579 and 1.439 to 1.403 respectively, surpassing existing competitive methods. Our codes are available at:https://github.com/CVEO/S3Net.

翻译：立体匹配与语义分割是双目卫星三维重建中的关键任务，然而以往研究主要将二者视为独立平行任务，缺乏集成化的多任务学习框架。本文提出一种单分支语义立体网络（Single-branch Semantic Stereo Network, S3Net），通过自融合与互融合模块创新性地将语义分割与立体匹配相结合。不同于先前面法孤立使用语义或视差信息的方式，本方法识别并利用这两个任务间的内在关联，从而实现对语义信息与视差估计的更精确理解。在US3D数据集上的对比测试验证了S3Net的有效性：语义分割的mIoU从61.38提升至67.39，视差估计的D1-误差与平均端点误差分别从10.051降至9.579、1.439降至1.403，全面超越现有竞争方法。代码已开源：https://github.com/CVEO/S3Net。