Scene understanding based on image segmentation is a crucial component for autonomous vehicles. Pixel-wise semantic segmentation of RGB images can be advanced by exploiting informative features from the supplementary modality (X-modality). In this work, we propose CMX, a transformer-based cross-modal fusion framework for RGB-X semantic segmentation. To generalize to different sensing modalities encompassing various supplements and uncertainties, we consider that comprehensive cross-modal interactions should be provided. CMX is built with two streams to extract features from RGB images and the X-modality. In each feature extraction stage, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate the feature of the current modality by combining the feature from the other modality, in spatial- and channel-wise dimensions. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to mix them for the final semantic prediction. FFM is constructed with a cross-attention mechanism, which enables exchange of long-range contexts, enhancing bi-modal features globally. Extensive experiments show that CMX generalizes to diverse multi-modal combinations, achieving state-of-the-art performances on five RGB-Depth benchmarks, as well as RGB-Thermal, RGB-Polarization, and RGB-LiDAR datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish an RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. The source code of CMX is publicly available at https://github.com/huaaaliu/RGBX_Semantic_Segmentation.

翻译：基于图像分割的场景理解是自动驾驶系统中的关键组成部分。通过利用辅助模态（X模态）的信息特征，可以提升RGB图像的像素级语义分割性能。本文提出CMX——一种基于Transformer的RGB-X跨模态融合框架，用于语义分割。为泛化至包含不同补充信息和不确定性的多种传感模态，我们考虑应提供全面的跨模态交互机制。CMX采用双流架构分别从RGB图像和X模态中提取特征。在每个特征提取阶段，我们设计跨模态特征校正模块（CM-FRM），通过融合另一模态的特征，从空间维度和通道维度对当前模态特征进行校准。基于校正后的特征对，我们部署特征融合模块（FFM）进行混合处理，以生成最终语义预测。FFM采用交叉注意力机制构建，能够实现长距离上下文信息的交换，从而全局增强双模态特征。大量实验表明，CMX可有效泛化至多种多模态组合，在五个RGB-深度基准数据集以及RGB-热红外、RGB-偏振和RGB-激光雷达数据集上均取得最优性能。此外，为探究稠密-稀疏数据融合的泛化能力，我们基于EventScape数据集建立了RGB-事件语义分割基准，CMX在该基准上创造了新的最优结果。CMX源代码已公开于https://github.com/huaaaliu/RGBX_Semantic_Segmentation。