As a popular paradigm of distributed learning, personalized federated learning (PFL) allows personalized models to improve generalization ability and robustness by utilizing knowledge from all distributed clients. Most existing PFL algorithms tackle personalization in a model-centric way, such as personalized layer partition, model regularization, and model interpolation, which all fail to take into account the data characteristics of distributed clients. In this paper, we propose a novel PFL framework for image classification tasks, dubbed pFedPT, that leverages personalized visual prompts to implicitly represent local data distribution information of clients and provides that information to the aggregation model to help with classification tasks. Specifically, in each round of pFedPT training, each client generates a local personalized prompt related to local data distribution. Then, the local model is trained on the input composed of raw data and a visual prompt to learn the distribution information contained in the prompt. During model testing, the aggregated model obtains prior knowledge of the data distributions based on the prompts, which can be seen as an adaptive fine-tuning of the aggregation model to improve model performances on different clients. Furthermore, the visual prompt can be added as an orthogonal method to implement personalization on the client for existing FL methods to boost their performance. Experiments on the CIFAR10 and CIFAR100 datasets show that pFedPT outperforms several state-of-the-art (SOTA) PFL algorithms by a large margin in various settings.

翻译：作为分布式学习的一种流行范式，个性化联邦学习（PFL）允许通过利用所有分布式客户端的知识来提升个性化模型的泛化能力和鲁棒性。现有的大多数PFL算法以模型为中心的方式处理个性化问题，例如个性化层划分、模型正则化和模型插值，但这些方法均未考虑分布式客户端的数据特征。在本文中，我们针对图像分类任务提出了一种新颖的PFL框架，称为pFedPT，该框架利用个性化视觉提示隐式表示客户端的本地数据分布信息，并将该信息提供给聚合模型以辅助分类任务。具体而言，在pFedPT训练的每一轮中，每个客户端生成与本地数据分布相关的个性化提示。然后，本地模型在由原始数据和视觉提示组成的输入上进行训练，以学习提示中包含的分布信息。在模型测试阶段，聚合模型基于提示获得数据分布的先验知识，这可以看作是对聚合模型的自适应微调，从而提升模型在不同客户端上的性能。此外，视觉提示可作为正交方法添加，用于在客户端上实现个性化，以提升现有联邦学习方法的性能。在CIFAR10和CIFAR100数据集上的实验表明，pFedPT在各种设置下均显著优于多种最先进的（SOTA）PFL算法。