Federated fine-tuning for Large Language Models (LLMs) has recently gained attention due to the heavy communication overhead of transmitting large model updates. Low Rank Adaptation (LoRA) has been proposed as a solution, yet its application in federated learning is complicated by discordance in aggregation. Existing methods addressing this discordance often suffer from performance degradation at low ranks in heterogeneous data settings. In response, we introduce LoRA-A2 (Low Rank Adaptation with Alternating freeze and Adaptive rank selection), which demonstrates robustness in challenging settings with low ranks and high data heterogeneity. Our experimental findings reveal that LoRA-A2 maintains performance even under extreme heterogeneity and low rank conditions, achieving up to a 99.8% reduction in uploaded parameters compared to full fine-tuning without compromising performance. This adaptive mechanism boosts robustness and communication efficiency in federated fine-tuning, enabling the practical deployment of LLMs in resource-constrained environments.

翻译：大型语言模型（LLM）的联邦微调因传输大规模模型更新带来的沉重通信开销而近期受到关注。低秩适应（LoRA）被提出作为一种解决方案，但其在联邦学习中的应用因聚合过程中的不协调性而变得复杂。现有解决这种不协调性的方法在异构数据设置下常面临低秩时性能下降的问题。为此，我们提出了LoRA-A2（具有交替冻结与自适应秩选择的低秩适应方法），该方法在低秩和高数据异质性的挑战性设置中展现出鲁棒性。我们的实验结果表明，即使在极端异质性和低秩条件下，LoRA-A2仍能保持性能，相比全参数微调，上传参数量减少高达99.8%，且性能无损。这种自适应机制增强了联邦微调的鲁棒性和通信效率，使得LLM在资源受限环境中的实际部署成为可能。