The increasing availability of multi-sensor data sparks interest in multimodal self-supervised learning. However, most existing approaches learn only common representations across modalities while ignoring intra-modal training and modality-unique representations. We propose Decoupling Common and Unique Representations (DeCUR), a simple yet effective method for multimodal self-supervised learning. By distinguishing inter- and intra-modal embeddings, DeCUR is trained to integrate complementary information across different modalities. We evaluate DeCUR in three common multimodal scenarios (radar-optical, RGB-elevation, and RGB-depth), and demonstrate its consistent benefits on scene classification and semantic segmentation downstream tasks. Notably, we get straightforward improvements by transferring our pretrained backbones to state-of-the-art supervised multimodal methods without any hyperparameter tuning. Furthermore, we conduct a comprehensive explainability analysis to shed light on the interpretation of common and unique features in our multimodal approach. Codes are available at \url{https://github.com/zhu-xlab/DeCUR}.

翻译：多传感器数据的日益普及激发了人们对多模态自监督学习的兴趣。然而，现有方法大多仅学习模态间的共有表征，忽略了模态内训练及模态独有表征。我们提出解耦共有与独有表征（DeCUR）方法，这是一种简单而有效的多模态自监督学习方法。通过区分模态间与模态内嵌入，DeCUR能整合不同模态间的互补信息。我们在三种常见的多模态场景（雷达-光学、RGB-高程、RGB-深度）中评估了DeCUR，并验证了其在场景分类和语义分割下游任务中持续稳定的性能提升。值得注意的是，在无需任何超参数调优的情况下，通过将预训练主干网络迁移至最先进的监督多模态方法，我们获得了直接性的性能改进。此外，我们进行了全面的可解释性分析，以阐明多模态方法中共有与独有特征的解析机制。代码开源地址：\url{https://github.com/zhu-xlab/DeCUR}。