With the increasing availability of Foundation Models, federated tuning has garnered attention in the field of federated learning, utilizing data and computation resources from multiple clients to collaboratively fine-tune foundation models. However, in real-world federated scenarios, there often exist a multitude of heterogeneous clients with varying computation and communication resources, rendering them incapable of supporting the entire model fine-tuning process. In response to this challenge, we propose a novel federated tuning algorithm, FedRA. The implementation of FedRA is straightforward and can be seamlessly integrated into any transformer-based model without the need for further modification to the original model. Specifically, in each communication round, FedRA randomly generates an allocation matrix. For resource-constrained clients, it reorganizes a small number of layers from the original model based on the allocation matrix and fine-tunes using LoRA. Subsequently, the server aggregates the updated LoRA parameters from the clients according to the current allocation matrix into the corresponding layers of the original model. It is worth noting that FedRA also supports scenarios where none of the clients can support the entire global model, which is an impressive advantage. We conduct experiments on two large-scale image datasets, DomainNet and NICO++, under various non-iid settings. The results demonstrate that FedRA outperforms the compared methods significantly. The source code is available at \url{https://github.com/leondada/FedRA}.

翻译：随着基础模型的日益普及，联邦调优在联邦学习领域受到广泛关注，它利用多个客户端的数据和计算资源协同微调基础模型。然而，在现实的联邦场景中，常存在大量计算与通信资源各异的异构客户端，使其无法支持完整的模型微调过程。针对这一挑战，我们提出了一种新颖的联邦调优算法FedRA。FedRA的实现简单直接，可无缝集成到任何基于Transformer的模型中，无需对原模型进行额外修改。具体而言，在每个通信轮次中，FedRA随机生成一个分配矩阵。对于资源受限的客户端，它基于该分配矩阵从原模型中重组少量层，并使用LoRA进行微调。随后，服务器根据当前分配矩阵将客户端更新的LoRA参数聚合到原模型的对应层中。值得注意的是，FedRA还支持所有客户端均无法承载完整全局模型的场景，这是一个令人瞩目的优势。我们在两个大规模图像数据集DomainNet和NICO++上进行了多种非独立同分布设置下的实验。结果表明，FedRA的性能显著优于对比方法。源代码已公开在\url{https://github.com/leondada/FedRA}。