Federated low-rank adaptation (FedLoRA) has facilitated communication-efficient and privacy-preserving fine-tuning of foundation models for downstream tasks. In practical federated learning scenarios, client heterogeneity in system resources and data distributions motivates heterogeneous LoRA ranks across clients. We identify a previously overlooked phenomenon in heterogeneous FedLoRA, termed rank collapse, where the energy of the global update concentrates on the minimum shared rank, resulting in suboptimal performance and high sensitivity to rank configurations. Through theoretical analysis, we reveal the root cause of rank collapse: a mismatch between rank-agnostic aggregation weights and rank-dependent client contributions, which systematically suppresses higher-rank updates at a geometric rate over rounds. Motivated by this insight, we propose raFLoRA, a rank-partitioned aggregation method that decomposes local updates into rank partitions and then aggregates each partition weighted by its effective client contributions. Extensive experiments across classification and reasoning tasks show that raFLoRA prevents rank collapse, improves model performance, and preserves communication efficiency compared to state-of-the-art FedLoRA baselines.

翻译：联邦低秩自适应（FedLoRA）促进了基础模型在下游任务中进行通信高效且保护隐私的微调。在实际的联邦学习场景中，客户端在系统资源和数据分布上的异构性促使了跨客户端的异构LoRA秩。我们发现了在异构FedLoRA中一个先前被忽视的现象，称为秩崩溃，即全局更新的能量集中在最小共享秩上，导致次优性能和对秩配置的高度敏感性。通过理论分析，我们揭示了秩崩溃的根本原因：秩无关的聚合权重与秩相关的客户端贡献之间的不匹配，这会在多轮迭代中以几何速率系统地抑制更高秩的更新。基于这一洞见，我们提出了raFLoRA，一种秩分区聚合方法，该方法将局部更新分解为秩分区，然后根据每个分区的有效客户端贡献进行加权聚合。在分类和推理任务上的大量实验表明，与最先进的FedLoRA基线相比，raFLoRA能够防止秩崩溃，提高模型性能，并保持通信效率。