Multimodal federated learning (FL) aims to enrich model training in FL settings where clients are collecting measurements across multiple modalities. However, key challenges to multimodal FL remain unaddressed, particularly in heterogeneous network settings where: (i) the set of modalities collected by each client will be diverse, and (ii) communication limitations prevent clients from uploading all their locally trained modality models to the server. In this paper, we propose multimodal Federated learning with joint Modality and Client selection (mmFedMC), a new FL methodology that can tackle the above-mentioned challenges in multimodal settings. The joint selection algorithm incorporates two main components: (a) A modality selection methodology for each client, which weighs (i) the impact of the modality, gauged by Shapley value analysis, (ii) the modality model size as a gauge of communication overhead, against (iii) the frequency of modality model updates, denoted recency, to enhance generalizability. (b) A client selection strategy for the server based on the local loss of modality model at each client. Experiments on five real-world datasets demonstrate the ability of mmFedMC to achieve comparable accuracy to several baselines while reducing the communication overhead by over 20x. A demo video of our methodology is available at https://liangqiy.com/mmfedmc/.

翻译：多模态联邦学习旨在丰富联邦学习环境中的模型训练，在该环境中，客户端跨多种模态收集测量数据。然而，多模态联邦学习的关键挑战仍未得到解决，尤其是在异构网络环境中：（i）每个客户端收集的模态集合具有多样性，（ii）通信限制使得客户端无法将所有本地训练的模态模型上传至服务器。本文提出了一种带联合模态与客户端选择机制的多模态联邦学习方法（mmFedMC），这是一种能够应对多模态环境中上述挑战的新型联邦学习方法。该联合选择算法包含两个主要部分：（a）为每个客户端设计的模态选择方法，该方法权衡以下因素：（i）基于沙普利值分析的模态影响程度，（ii）衡量通信开销的模态模型大小，以及（iii）用于增强泛化能力的模态模型更新频率（即新近度）；（b）基于每个客户端模态模型的局部损失为服务器设计的客户端选择策略。在五个真实世界数据集上的实验表明，mmFedMC能够在将通信开销降低20倍以上的同时，实现与多种基线方法相当的精度。相关方法演示视频可访问https://liangqiy.com/mmfedmc/。