Federated learning (FL) enables multiple clients to collaboratively train a global model without disclosing their data. Previous researches often require training the complete model parameters. However, the emergence of powerful pre-trained models makes it possible to achieve higher performance with fewer learnable parameters in FL. In this paper, we propose a federated adaptive prompt tuning algorithm, FedAPT, for multi-domain collaborative image classification with powerful foundation models, like CLIP. Compared with direct federated prompt tuning, our core idea is to adaptively unlock specific domain knowledge for each test sample in order to provide them with personalized prompts. To implement this idea, we design an adaptive prompt tuning module, which consists of a meta prompt, an adaptive network, and some keys. The server randomly generates a set of keys and assigns a unique key to each client. Then all clients cooperatively train the global adaptive network and meta prompt with the local datasets and the frozen keys. Ultimately, the global aggregation model can assign a personalized prompt to CLIP based on the domain features of each test sample. We perform extensive experiments on two multi-domain image classification datasets across two different settings -- supervised and unsupervised. The results show that FedAPT can achieve better performance with less than 10\% of the number of parameters of the fully trained model, and the global model can perform well in diverse client domains simultaneously.

翻译：联邦学习使多个客户端能够在不泄露数据的情况下协作训练全局模型。以往研究通常需要训练完整模型参数。然而，强大的预训练模型的出现使得在联邦学习中用更少可学习参数实现更高性能成为可能。本文提出一种联邦自适应提示调优算法FedAPT，用于基于CLIP等强大基础模型的多域协作图像分类。与直接联邦提示调优相比，我们的核心思想是为每个测试样本自适应解锁特定领域知识，为其提供个性化提示。为实现这一思想，我们设计了一个自适应提示调优模块，该模块由元提示、自适应网络和一些密钥组成。服务器随机生成一组密钥，并为每个客户端分配唯一密钥。随后所有客户端利用本地数据集和冻结的密钥协同训练全局自适应网络和元提示。最终，全局聚合模型能够根据每个测试样本的域特征为CLIP分配个性化提示。我们在两个多域图像分类数据集上进行了监督和无监督两种设置下的广泛实验。结果表明，FedAPT能以不到完全训练模型10%的参数数量实现更优性能，且全局模型能同时在多个客户端域中良好运行。