Federated learning (FL) emerges as a decentralized learning framework which trains models from multiple distributed clients without sharing their data to preserve privacy. Recently, large-scale pre-trained models (e.g., Vision Transformer) have shown a strong capability of deriving robust representations. However, the data heterogeneity among clients, the limited computation resources, and the communication bandwidth restrict the deployment of large-scale models in FL frameworks. To leverage robust representations from large-scale models while enabling efficient model personalization for heterogeneous clients, we propose a novel personalized FL framework of client-specific Prompt Generation (pFedPG), which learns to deploy a personalized prompt generator at the server for producing client-specific visual prompts that efficiently adapts frozen backbones to local data distributions. Our proposed framework jointly optimizes the stages of personalized prompt adaptation locally and personalized prompt generation globally. The former aims to train visual prompts that adapt foundation models to each client, while the latter observes local optimization directions to generate personalized prompts for all clients. Through extensive experiments on benchmark datasets, we show that our pFedPG is favorable against state-of-the-art personalized FL methods under various types of data heterogeneity, allowing computation and communication efficient model personalization.

翻译：联邦学习作为一种去中心化学习框架，通过在不共享数据的前提下从多个分布式客户端训练模型以保护隐私。近年来，大规模预训练模型（如视觉Transformer）展现出提取鲁棒表示的强大能力。然而，客户端间的数据异质性、有限的计算资源以及通信带宽限制了大模型在联邦学习框架中的部署。为利用大规模模型的鲁棒表示，同时实现面向异构客户端的高效模型个性化，我们提出了一种新颖的个性化联邦学习框架——客户端特定提示生成（pFedPG）。该框架学习在服务器端部署个性化提示生成器，用于生成客户端特定的视觉提示，从而高效地将冻结主干网络适配至本地数据分布。所提出的框架联合优化了本地个性化提示适配与全局个性化提示生成两个阶段：前者旨在训练使基础模型适配每个客户端的视觉提示，后者则通过观察本地优化方向为所有客户端生成个性化提示。通过在基准数据集上的大量实验，我们证明pFedPG在各类数据异质性条件下优于现有最先进的个性化联邦学习方法，实现了计算与通信高效的模型个性化。