Federated learning (FL) is an emerging machine learning paradigm in which a central server coordinates multiple participants (clients) collaboratively to train on decentralized data. In practice, FL often faces statistical, system, and model heterogeneities, which inspires the field of Model-Heterogeneous Personalized Federated Learning (MHPFL). With the increased interest in adopting large language models (LLMs) in FL, the existing MHPFL methods cannot achieve acceptable computational and communication costs, while maintaining satisfactory model performance. To bridge this gap, we propose a novel and efficient model-heterogeneous personalized Federated learning framework based on LoRA tuning (pFedLoRA). Inspired by the popular LoRA method for fine-tuning pre-trained LLMs with a low-rank model (a.k.a., an adapter), we design a homogeneous small adapter to facilitate federated client's heterogeneous local model training with our proposed iterative training for global-local knowledge exchange. The homogeneous small local adapters are aggregated on the FL server to generate a global adapter. We theoretically prove the convergence of pFedLoRA. Extensive experiments on two benchmark datasets demonstrate that pFedLoRA outperforms six state-of-the-art baselines, beating the best method by 1.35% in test accuracy, 11.81 times computation overhead reduction and 7.41 times communication cost saving.

翻译：联邦学习（FL）是一种新兴的机器学习范式，其中中央服务器协调多个参与者（客户端）协作训练分散数据。在实际应用中，FL常面临统计异构性、系统异构性和模型异构性，这催生了模型异构个性化联邦学习（MHPFL）领域。随着将大型语言模型（LLMs）引入FL的需求日益增长，现有MHPFL方法无法在保持满意模型性能的同时，实现可接受的计算和通信成本。为填补这一空白，我们提出了一种新颖且高效的基于LoRA微调的模型异构个性化联邦学习框架（pFedLoRA）。受流行的LoRA方法（通过低秩模型（即适配器）微调预训练LLM）的启发，我们设计了一种同构的小型适配器，通过提出的全局-局部知识交换迭代训练方法，促进联邦客户端异构本地模型的训练。同构的小型本地适配器在FL服务器上聚合，生成全局适配器。我们从理论上证明了pFedLoRA的收敛性。在两个基准数据集上的大量实验表明，pFedLoRA优于六种最先进的基线方法，在测试准确率上比最佳方法提升1.35%，计算开销降低11.81倍，通信成本节省7.41倍。