Recently, foundation models have exhibited remarkable advancements in multi-modal learning. These models, equipped with millions (or billions) of parameters, typically require a substantial amount of data for finetuning. However, collecting and centralizing training data from diverse sectors becomes challenging due to distinct privacy regulations. Federated Learning (FL) emerges as a promising solution, enabling multiple clients to collaboratively train neural networks without centralizing their local data. To alleviate client computation burdens and communication overheads, previous works have adapted Parameter-efficient Finetuning (PEFT) methods for FL. Hereby, only a small fraction of the model parameters are optimized and communicated during federated communications. Nevertheless, most previous works have focused on a single modality and neglected one common phenomenon, i.e., the presence of data heterogeneity across the clients. Therefore, in this work, we propose a finetuning framework tailored to heterogeneous multi-modal FL, called Federated Dual-Aadapter Teacher (FedDAT). Specifically, our approach leverages a Dual-Adapter Teacher (DAT) to address data heterogeneity by regularizing the client local updates and applying Mutual Knowledge Distillation (MKD) for an efficient knowledge transfer. FedDAT is the first approach that enables an efficient distributed finetuning of foundation models for a variety of heterogeneous Vision-Language tasks. To demonstrate its effectiveness, we conduct extensive experiments on four multi-modality FL benchmarks with different types of data heterogeneity, where FedDAT substantially outperforms the existing centralized PEFT methods adapted for FL.

翻译：近期，基础模型在多模态学习领域取得了显著进展。这些模型拥有数百万（或数十亿）参数，通常需要大量数据用于微调。然而，由于各行业存在严格的隐私法规，收集并集中训练数据面临巨大挑战。联邦学习作为一种有前景的解决方案应运而生，它允许多个客户端在不集中本地数据的情况下协作训练神经网络。为减轻客户端计算负担与通信开销，先前研究已将参数高效微调方法适配至联邦学习场景。在此框架下，联邦通信过程中仅需优化和传输模型参数的极小部分。然而，大多数前期研究聚焦于单一模态，忽视了跨客户端数据异构这一普遍现象。为此，本文提出一种面向异构多模态联邦学习的微调框架——联邦双适配器教师（FedDAT）。具体而言，该方法通过双适配器教师（DAT）正则化客户端本地更新以应对数据异构性，并利用互知识蒸馏（MKD）实现高效知识迁移。FedDAT首次实现了针对多种异构视觉-语言任务的基础模型高效分布式微调。为验证其有效性，我们在四种具有不同类型数据异构性的多模态联邦学习基准上开展了大量实验，结果表明FedDAT显著优于现有面向联邦学习适配的集中式参数高效微调方法。