Federated learning systems have been identified as an efficient approach to scaling distributed model training with a large amount of participants or data owners while guaranteeing data privacy. To apply the current most popular pre-trained large language models to other domains with data privacy guarantee requirements, existing works propose fine-tuning the pre-trained large language models in federated learning environments across data owners using the parameter efficient fine-tuning approaches, LoRA. To address the resource and data heterogeneous issues for the participants, previous works adopted heterogeneous LoRA using different ranks for different clients and pending their rank, which brings bias for the parameter aggregation. To address this issue, we propose HLoRA, an efficient federated learning system utilizing a modified LoRA approach that incorporates rank heterogeneity to optimize communication and computational efficiency. Experimental results, conducted using the Microsoft Research Paraphrase Corpus (MRPC), Quora Question Pairs (QQP) and Recognizing Textual Entailment (RTE), within the Plato federated learning framework, demonstrate that our method not only reduces resource demands but also outperforms traditional LoRA applications in terms of convergence speed and final model accuracy. This study shows that our approach can significantly improve the practical deployment of federated LLM fine-tuning, particularly in environments with diverse client resources.

翻译：联邦学习系统已被证实为一种高效方法，可在保障数据隐私的前提下，利用大量参与者或数据所有者扩展分布式模型训练。为将当前最流行的预训练大语言模型应用于其他具有数据隐私保障需求的领域，现有研究提出在联邦学习环境中，利用参数高效微调方法LoRA，跨数据所有者对预训练大语言模型进行微调。为解决参与者的资源与数据异构性问题，先前工作采用异构LoRA，为不同客户端分配不同秩并聚合其秩，但这会引入参数聚合偏差。为解决此问题，我们提出HLoRA——一种高效的联邦学习系统，它采用改进的LoRA方法，通过纳入秩异构性来优化通信与计算效率。在Plato联邦学习框架内，使用微软研究复述语料库（MRPC）、Quora问题对（QQP）和文本蕴含识别（RTE）数据集进行的实验结果表明，我们的方法不仅降低了资源需求，而且在收敛速度和最终模型准确率方面均优于传统LoRA应用。本研究表明，我们的方法能显著提升联邦大语言模型微调的实际部署效果，尤其在客户端资源多样化的环境中。