Over the past few years, Federated Learning (FL) has become an emerging machine learning technique to tackle data privacy challenges through collaborative training. In the Federated Learning algorithm, the clients submit a locally trained model, and the server aggregates these parameters until convergence. Despite significant efforts that have been made to FL in fields like computer vision, audio, and natural language processing, the FL applications utilizing multimodal data streams remain largely unexplored. It is known that multimodal learning has broad real-world applications in emotion recognition, healthcare, multimedia, and social media, while user privacy persists as a critical concern. Specifically, there are no existing FL benchmarks targeting multimodal applications or related tasks. In order to facilitate the research in multimodal FL, we introduce FedMultimodal, the first FL benchmark for multimodal learning covering five representative multimodal applications from ten commonly used datasets with a total of eight unique modalities. FedMultimodal offers a systematic FL pipeline, enabling end-to-end modeling framework ranging from data partition and feature extraction to FL benchmark algorithms and model evaluation. Unlike existing FL benchmarks, FedMultimodal provides a standardized approach to assess the robustness of FL against three common data corruptions in real-life multimodal applications: missing modalities, missing labels, and erroneous labels. We hope that FedMultimodal can accelerate numerous future research directions, including designing multimodal FL algorithms toward extreme data heterogeneity, robustness multimodal FL, and efficient multimodal FL. The datasets and benchmark results can be accessed at: https://github.com/usc-sail/fed-multimodal.

翻译：近年来，联邦学习（FL）已成为一种新兴的机器学习技术，通过协同训练应对数据隐私挑战。在联邦学习算法中，客户端提交本地训练的模型，服务器聚合这些参数直至收敛。尽管联邦学习在计算机视觉、音频和自然语言处理等领域已取得显著进展，但利用多模态数据流的联邦学习应用仍鲜有探索。众所周知，多模态学习在情感识别、医疗健康、多媒体和社交媒体等领域具有广泛的现实应用，而用户隐私始终是关键问题。具体而言，目前尚无针对多模态应用或相关任务的现有联邦学习基准测试。为促进多模态联邦学习的研究，我们提出了FedMultimodal——首个面向多模态学习的联邦学习基准测试，涵盖来自十个常用数据集的五种代表性多模态应用，共涉及八种独特模态。FedMultimodal提供系统化的联邦学习流水线，支持从数据划分、特征提取到联邦学习基准算法和模型评估的端到端建模框架。与现有联邦学习基准不同，FedMultimodal提供标准化方法，用于评估联邦学习在现实多模态应用中针对三种常见数据损坏（模态缺失、标签缺失和标签错误）的鲁棒性。我们希望FedMultimodal能够加速多项未来研究方向，包括设计面向极端数据异质性的多模态联邦学习算法、鲁棒性多模态联邦学习以及高效多模态联邦学习。数据集和基准测试结果可通过以下链接获取：https://github.com/usc-sail/fed-multimodal。