Federated learning (FL) has emerged as a new paradigm for privacy-preserving computation in recent years. Unfortunately, FL faces two critical challenges that hinder its actual performance: data distribution heterogeneity and high resource costs brought by large foundation models. Specifically, the non-IID data in different clients make existing FL algorithms hard to converge while the high resource costs, including computational and communication costs that increase the deployment difficulty in real-world scenarios. In this paper, we propose an effective yet simple method, named FedCLIP, to achieve fast generalization and personalization for CLIP in federated learning. Concretely, we design an attention-based adapter for the large model, CLIP, and the rest operations merely depend on adapters. Lightweight adapters can make the most use of pretrained model information and ensure models be adaptive for clients in specific tasks. Simultaneously, small-scale operations can mitigate the computational burden and communication burden caused by large models. Extensive experiments are conducted on three datasets with distribution shifts. Qualitative and quantitative results demonstrate that FedCLIP significantly outperforms other baselines (9% overall improvements on PACS) and effectively reduces computational and communication costs (283x faster than FedAVG). Our code will be available at: https://github.com/microsoft/PersonalizedFL.

翻译：联邦学习（FL）近年来已成为一种隐私保护计算的新范式。然而，FL面临两大关键挑战，制约了其实际性能：数据分布异质性和大规模基础模型带来的高资源成本。具体而言，不同客户端上的非独立同分布数据使得现有FL算法难以收敛，而高资源成本（包括计算与通信成本）则增加了实际场景中的部署难度。本文提出一种称为FedCLIP的有效且简洁的方法，旨在实现联邦学习中CLIP模型的快速泛化与个性化。具体地，我们为大规模模型CLIP设计了一种基于注意力的适配器，其余操作仅依赖于该适配器。轻量级适配器能充分利用预训练模型信息，并确保模型适应客户端的特定任务。同时，小规模操作可缓解大规模模型带来的计算与通信负担。我们在三个存在分布偏移的数据集上进行了广泛实验。定性与定量结果表明，FedCLIP显著优于其他基线方法（在PACS上总体提升9%），并有效降低了计算与通信成本（比FedAVG快283倍）。我们的代码将在https://github.com/microsoft/PersonalizedFL 公开。