Selecting proper clients to participate in the iterative federated learning (FL) rounds is critical to effectively harness a broad range of distributed datasets. Existing client selection methods simply consider the variability among FL clients with uni-modal data, however, have yet to consider clients with multi-modalities. We reveal that traditional client selection scheme in MFL may suffer from a severe modality-level bias, which impedes the collaborative exploitation of multi-modal data, leading to insufficient local data exploration and global aggregation. To tackle this challenge, we propose a Client-wise Modality Selection scheme for MFL (CMSFed) that can comprehensively utilize information from each modality via avoiding such client selection bias caused by modality imbalance. Specifically, in each MFL round, the local data from different modalities are selectively employed to participate in local training and aggregation to mitigate potential modality imbalance of the global model. To approximate the fully aggregated model update in a balanced way, we introduce a novel local training loss function to enhance the weak modality and align the divergent feature spaces caused by inconsistent modality adoption strategies for different clients simultaneously. Then, a modality-level gradient decoupling method is designed to derive respective submodular functions to maintain the gradient diversity during the selection progress and balance MFL according to local modality imbalance in each iteration. Our extensive experiments showcase the superiority of CMSFed over baselines and its effectiveness in multi-modal data exploitation.

翻译：在迭代式联邦学习（FL）轮次中，选择合适的客户端参与训练对于有效利用广泛分布的数据集至关重要。现有的客户端选择方法仅考虑了单模态数据下FL客户端间的差异性，尚未顾及多模态数据下的客户端场景。我们发现，传统客户端选择方案在多模态联邦学习（MFL）中可能面临严重的模态级偏差，这会阻碍多模态数据的协同利用，导致局部数据探索不足与全局聚合效果不佳。为攻克这一难题，我们提出了一种面向MFL的客户端模态选择方案（CMSFed），通过避免模态不平衡所引发的客户端选择偏差，实现对每种模态信息的全面利用。具体而言，在每个MFL轮次中，我们有选择性地采用来自不同模态的局部数据参与本地训练与聚合，以缓解全局模型潜在的模态不平衡。为以均衡方式近似完全聚合的模型更新，我们引入了一种新型局部训练损失函数，既能增强弱模态能力，又能同步对齐因不同客户端采用不一致模态采纳策略而导致的特征空间差异。随后，设计了一种模态级梯度解耦方法，推导出各自对应的子模函数，在选拔过程中维持梯度多样性，并根据每次迭代中局部模态不平衡状态平衡MFL。大量实验证明，CMSFed相较于基线方法具有优越性，并在多模态数据利用方面展现出显著效果。