In this paper, we introduce a novel Synchronized Class Token Fusion (SCT Fusion) architecture in the framework of multi-modal multi-label classification (MLC) of remote sensing (RS) images. The proposed architecture leverages modality-specific attention-based transformer encoders to process varying input modalities, while exchanging information across modalities by synchronizing the special class tokens after each transformer encoder block. The synchronization involves fusing the class tokens with a trainable fusion transformation, resulting in a synchronized class token that contains information from all modalities. As the fusion transformation is trainable, it allows to reach an accurate representation of the shared features among different modalities. Experimental results show the effectiveness of the proposed architecture over single-modality architectures and an early fusion multi-modal architecture when evaluated on a multi-modal MLC dataset. The code of the proposed architecture is publicly available at https://git.tu-berlin.de/rsim/sct-fusion.

翻译：本文提出了一种新颖的同步类别令牌融合（SCT Fusion）架构，应用于遥感图像的多模态多标签分类（MLC）框架中。该架构利用基于注意力机制的多模态特定Transformer编码器处理不同输入模态，并通过在每个Transformer编码器块后同步特殊类别令牌来实现模态间的信息交换。同步过程涉及利用可训练的融合变换对类别令牌进行融合，从而生成包含所有模态信息的同步类别令牌。由于融合变换是可训练的，它能够精确表示不同模态间的共享特征。实验结果表明，在多模态MLC数据集上评估时，所提架构相较于单模态架构及早期融合的多模态架构具有有效性。所提架构的代码已在https://git.tu-berlin.de/rsim/sct-fusion 公开提供。