The development of federated learning (FL) methods, which aim to learn from distributed databases (i.e., clients) without accessing data on clients, has recently attracted great attention. Most of these methods assume that the clients are associated with the same data modality. However, remote sensing (RS) images in different clients can be associated with different data modalities that can improve the classification performance when jointly used. To address this problem, in this paper we introduce a novel multi-modal FL framework that aims to learn from decentralized multi-modal RS image archives for RS image classification problems. The proposed framework is made up of three modules: 1) multi-modal fusion (MF); 2) feature whitening (FW); and 3) mutual information maximization (MIM). The MF module performs iterative model averaging to learn without accessing data on clients in the case that clients are associated with different data modalities. The FW module aligns the representations learned among the different clients. The MIM module maximizes the similarity of images from different modalities. Experimental results show the effectiveness of the proposed framework compared to iterative model averaging, which is a widely used algorithm in FL. The code of the proposed framework is publicly available at https://git.tu-berlin.de/rsim/MM-FL.

翻译：联邦学习（FL）方法旨在在不访问客户端数据的情况下从分布式数据库（即客户端）中学习，其发展近来引起了广泛关注。大多数现有方法假设客户端关联相同的数据模态。然而，不同客户端中的遥感（RS）图像可能关联不同的数据模态，当联合使用时可以提升分类性能。针对这一问题，本文提出了一种新颖的多模态联邦学习框架，旨在从去中心化的多模态遥感图像档案中学习以解决遥感图像分类问题。该框架由三个模块组成：1）多模态融合（MF）；2）特征白化（FW）；以及3）互信息最大化（MIM）。MF模块执行迭代模型平均，以在客户端关联不同数据模态的情况下实现不访问客户端数据的学习。FW模块对齐不同客户端学习到的表示。MIM模块最大化来自不同模态的图像之间的相似性。实验结果表明，与联邦学习中广泛使用的迭代模型平均算法相比，所提框架具有有效性。所提框架的代码已在 https://git.tu-berlin.de/rsim/MM-FL 公开。