Federated fine-tuning of large language models using parameter-efficient methods such as LoRA enables privacy-preserving adaptation of foundation models. Heterogeneous hardware resources introduce challenges, as clients with different adapter ranks cannot be directly aggregated. While existing methods enable aggregation under heterogeneous ranks, they fail to control how information is distributed across rank dimensions, leading to suboptimal use of shared low-rank representations. Instead, we propose PreLort: a nested low-rank formulation for federated LoRA that organizes adapter dimensions into a prefix hierarchy. Our approach ensures that lower-rank dimensions encode task-relevant information, while higher-rank dimensions capture additional capacity. Building on this, we introduce (i) a segment-wise aggregation rule that averages only over clients contributing to each rank segment, avoiding dilution from zero-padded lower-rank clients, and (ii) a prefix-nested training strategy that optimizes each adapter under multiple rank truncations, encouraging useful signal to concentrate in low-rank prefix dimensions. Together, these components encourage a consistent low-rank prefix capturing the most task-relevant information, while higher-rank dimensions learn additional capacity. This allows low-rank clients to benefit from richer information contributed by higher-rank clients, as prefix dimensions are consistently learned and aggregated. Experiments demonstrate that our method consistently outperforms prior heterogeneous federated LoRA methods in accuracy and ROUGE-L, while achieving lower or comparable perplexity across multiple base models.

翻译：联邦微调大型语言模型时，采用LoRA等参数高效方法可实现基础模型的隐私保护适配。异构硬件资源带来了挑战，因为具有不同适配器秩的客户端无法直接聚合。现有方法虽支持异构秩下的聚合，但无法控制信息在秩维度上的分布方式，导致共享低秩表示利用不充分。为此，我们提出PreLort：一种面向联邦LoRA的嵌套低秩公式，将适配器维度组织为前缀层级结构。该方法确保低秩维度编码任务相关信息，而高秩维度捕获额外容量。基于此，我们引入：（i）逐段聚合规则，仅对贡献于各秩段的客户端进行平均，避免来自零填充低秩客户端的稀释效应；（ii）前缀嵌套训练策略，在多个秩截断下优化每个适配器，促使有效信号集中于低秩前缀维度。这些组件协同作用，使低秩客户端能够从高秩客户端贡献的更丰富信息中获益，因为前缀维度被一致学习与聚合。实验表明，该方法在准确率和ROUGE-L指标上持续优于现有异构联邦LoRA方法，并在多个基座模型上达到更低或相当水平的困惑度。