Federated learning (FL) has emerged as a new paradigm for privacy-preserving computation in recent years. Unfortunately, FL faces two critical challenges that hinder its actual performance: data distribution heterogeneity and high resource costs brought by large foundation models. Specifically, the non-IID data in different clients make existing FL algorithms hard to converge while the high resource costs, including computational and communication costs that increase the deployment difficulty in real-world scenarios. In this paper, we propose an effective yet simple method, named FedCLIP, to achieve fast generalization and personalization for CLIP in federated learning. Concretely, we design an attention-based adapter for the large model, CLIP, and the rest operations merely depend on adapters. Lightweight adapters can make the most use of pretrained model information and ensure models be adaptive for clients in specific tasks. Simultaneously, small-scale operations can mitigate the computational burden and communication burden caused by large models. Extensive experiments are conducted on three datasets with distribution shifts. Qualitative and quantitative results demonstrate that FedCLIP significantly outperforms other baselines (9% overall improvements on PACS) and effectively reduces computational and communication costs (283x faster than FedAVG). Our code will be available at: https://github.com/microsoft/PersonalizedFL.

翻译：联邦学习（FL）近年来已成为隐私保护计算的新范式。然而，FL面临两个关键挑战制约其实际性能：数据分布异质性和大规模基础模型带来的高资源成本。具体而言，不同客户端上的非独立同分布数据使现有FL算法难以收敛，而高资源成本（包括计算和通信成本）则增加了实际场景中的部署难度。本文提出一种有效且简洁的方法——FedCLIP，旨在实现联邦学习中CLIP的快速泛化与个性化。具体地，我们为大型模型CLIP设计了一个基于注意力的适配器，其余操作仅依赖该适配器。轻量级适配器能够充分利用预训练模型信息，确保模型适应客户端的特定任务。同时，小规模操作可缓解大型模型带来的计算和通信负担。我们在三个具有分布偏移的数据集上进行了大量实验。定性和定量结果表明，FedCLIP显著优于其他基线方法（在PACS上整体提升9%），并有效降低了计算与通信成本（比FedAVG快283倍）。我们的代码将发布于：https://github.com/microsoft/PersonalizedFL。