With the increasing amount of multimedia data on modern mobile systems and IoT infrastructures, harnessing these rich multimodal data without breaching user privacy becomes a critical issue. Federated learning (FL) serves as a privacy-conscious alternative to centralized machine learning. However, existing FL methods extended to multimodal data all rely on model aggregation on single modality level, which restrains the server and clients to have identical model architecture for each modality. This limits the global model in terms of both model complexity and data capacity, not to mention task diversity. In this work, we propose Contrastive Representation Ensemble and Aggregation for Multimodal FL (CreamFL), a multimodal federated learning framework that enables training larger server models from clients with heterogeneous model architectures and data modalities, while only communicating knowledge on public dataset. To achieve better multimodal representation fusion, we design a global-local cross-modal ensemble strategy to aggregate client representations. To mitigate local model drift caused by two unprecedented heterogeneous factors stemming from multimodal discrepancy (modality gap and task gap), we further propose two inter-modal and intra-modal contrasts to regularize local training, which complements information of the absent modality for uni-modal clients and regularizes local clients to head towards global consensus. Thorough evaluations and ablation studies on image-text retrieval and visual question answering tasks showcase the superiority of CreamFL over state-of-the-art FL methods and its practical value.

翻译：随着现代移动系统和物联网基础设施中多媒体数据量的不断增加，如何在不侵犯用户隐私的前提下利用这些丰富的多模态数据成为关键问题。联邦学习（FL）作为集中式机器学习的隐私保护替代方案，但现有扩展至多模态数据的FL方法均依赖于单模态级别的模型聚合，这限制了服务器和客户端对每种模态采用相同的模型架构，从而在模型复杂度和数据容量方面制约了全局模型，更不用说任务多样性。本文提出基于对比表征集成与聚合的多模态联邦学习框架（CreamFL），该框架允许客户端使用异构模型架构和数据模态训练更大的服务器模型，同时仅在公共数据集上通信知识。为实现更好的多模态表征融合，我们设计了一种全局-局部跨模态集成策略来聚合客户端表征。针对多模态差异（模态间隙和任务间隙）导致的两种前所未有的异构因素引发的局部模型漂移，我们进一步提出两种模态间和模态内对比来正则化局部训练：为单模态客户端补充缺失模态的信息，并引导局部客户端向全局共识收敛。在图像-文本检索和视觉问答任务上的全面评估与消融实验展示了CreamFL相比现有最先进FL方法的优越性及其实际应用价值。