Large Language Models (LLMs) have demonstrated remarkable effectiveness in adapting to downstream tasks through fine-tuning. Federated Learning (FL) extends this capability by enabling collaborative fine-tuning across distributed clients using Low-Rank Adaptation (LoRA), while preserving data privacy by avoiding raw data sharing. However, practical deployments face challenges when clients have heterogeneous resources and thus adopt different LoRA ranks, leading to substantial initialization and aggregation noise that undermines performance. To address these challenges, we propose Fed-PLoRA, a novel lightweight heterogeneous federated fine-tuning (FFT) framework. Fed-PLoRA introduces Parallel One-Rank Adaptation (PLoRA), a new LoRA variant that replaces the classic multi-rank LoRA module with multiple parallel one-rank modules, and a novel Select-N-Fold strategy that folds untrained PLoRA modules into the pre-trained weights before local training, thereby accommodating heterogeneous client resources. We provide a unified analysis of initialization and aggregation noise of Fed-PLoRA and demonstrate how it addresses the limitations of state-of-the-art methods. Extensive experiments on diverse LLM fine-tuning tasks demonstrate that Fed-PLoRA consistently outperforms existing methods in both accuracy and efficiency. The code is available at https://github.com/TNI-playground/Fed-PLoRA.

翻译：大型语言模型（LLM）已证明通过微调能有效适应下游任务。联邦学习（FL）通过使用低秩适配（LoRA）在分布式客户端间实现协同微调，同时避免原始数据共享以保护数据隐私，从而扩展了此能力。然而，当客户端具有异构资源并因此采用不同LoRA秩时，实际部署面临挑战，这会导致显著的初始化和聚合噪声，从而损害性能。为解决这些挑战，我们提出Fed-PLoRA，一种新颖的轻量级异构联邦微调（FFT）框架。Fed-PLoRA引入了并行单秩适配（PLoRA），这是一种新的LoRA变体，用多个并行单秩模块替代经典的多秩LoRA模块，并提出新颖的Select-N-Fold策略，在本地训练前将未训练的PLoRA模块折叠到预训练权重中，从而适应异构客户端资源。我们对Fed-PLoRA的初始化和聚合噪声进行了统一分析，并展示了其如何解决现有最先进方法的局限性。在多样化的LLM微调任务上进行的大量实验表明，Fed-PLoRA在准确性和效率上均持续优于现有方法。代码发布于 https://github.com/TNI-playground/Fed-PLoRA。