With the increasing amount of multimedia data on modern mobile systems and IoT infrastructures, harnessing these rich multimodal data without breaching user privacy becomes a critical issue. Federated learning (FL) serves as a privacy-conscious alternative to centralized machine learning. However, existing FL methods extended to multimodal data all rely on model aggregation on single modality level, which restrains the server and clients to have identical model architecture for each modality. This limits the global model in terms of both model complexity and data capacity, not to mention task diversity. In this work, we propose Contrastive Representation Ensemble and Aggregation for Multimodal FL (CreamFL), a multimodal federated learning framework that enables training larger server models from clients with heterogeneous model architectures and data modalities, while only communicating knowledge on public dataset. To achieve better multimodal representation fusion, we design a global-local cross-modal ensemble strategy to aggregate client representations. To mitigate local model drift caused by two unprecedented heterogeneous factors stemming from multimodal discrepancy (modality gap and task gap), we further propose two inter-modal and intra-modal contrasts to regularize local training, which complements information of the absent modality for uni-modal clients and regularizes local clients to head towards global consensus. Thorough evaluations and ablation studies on image-text retrieval and visual question answering tasks showcase the superiority of CreamFL over state-of-the-art FL methods and its practical value.

翻译：随着现代移动系统和物联网基础设施上多媒体数据的日益增长，如何在不侵犯用户隐私的情况下利用这些丰富的多模态数据成为一个关键问题。联邦学习（FL）作为集中式机器学习的隐私保护替代方案，已被广泛采用。然而，现有扩展至多模态数据的联邦学习方法均依赖单一模态级别的模型聚合，这限制了服务器和客户端在每个模态上必须采用相同的模型架构，从而制约了全局模型在模型复杂度和数据容量方面的表现，更无法支持任务的多样性。本文提出了一种多模态联邦学习框架——对比表示集成与聚合的多模态联邦学习（CreamFL），该框架允许客户端使用异构模型架构和数据模态训练更大的服务器模型，同时仅通过公共数据集进行知识通信。为优化多模态表示融合，我们设计了一种全局-局部跨模态集成策略来聚合客户端表示。针对多模态差异（模态差距与任务差距）所导致的两个前所未有的异构因素引发的局部模型漂移问题，我们进一步提出了模态间与模态内的两种对比学习机制来正则化局部训练：对于单模态客户端，补充缺失模态的信息；同时引导局部客户端向全局共识方向优化。在图像-文本检索和视觉问答任务上的全面评估与消融实验表明，CreamFL优于当前最先进的联邦学习方法，并展现出其实用价值。