Federated learning (FL) for large language models (LLMs) has attracted increasing attention as a privacy-preserving approach for adapting models over distributed data, where parameter-efficient methods such as Low-Rank Adaptation (LoRA) are widely adopted to reduce communication and memory costs. However, practical deployments often exhibit rank and data heterogeneity: clients operate under different low-rank budgets and data distributions, making direct aggregation of LoRA updates biased and unstable. Existing approaches either enforce a unified rank or align heterogeneous updates into a single shared subspace, which tends to mix transferable and client-specific directions and consequently undermines personalization. Moreover, under differential privacy (DP), perturbing such structurally mixed updates injects noise into directions that should remain purely local, leading to unnecessary utility degradation. To address these issues, we propose Selective Decoupled Federated LoRA (SDFLoRA), a structure-aware LoRA framework that decouples each client update into a shared component for aggregation and a private component that preserves client-specific semantics. Only the shared component participates in subspace alignment, while the private component remains local and uncommunicated, making the training DP-compatible and stabilizing aggregation under rank heterogeneity. By injecting noise only into the aggregated shareable update, this approach avoids perturbations to local directions and improves the utility-privacy trade-off. Experiments on multiple benchmarks demonstrate that SDFLoRA outperforms federated LoRA baselines and achieves a strong utility-privacy trade-off.

翻译：联邦学习（FL）在大语言模型（LLM）领域作为适配分布式数据时保护隐私的方法日益受到关注，其中低秩适配（LoRA）等参数高效方法被广泛采用以降低通信和内存成本。然而，实际部署常面临秩与数据的双重异构性：客户端在低秩预算与数据分布不同的条件下运行，导致LoRA更新的直接聚合存在偏差且不稳定。现有方法要么强制统一秩，要么将异构更新对齐到单一共享子空间，这易混合可迁移方向与客户端特定方向，进而损害个性化能力。此外，在差分隐私（DP）场景中，扰动这类结构混合的更新会向本应保持纯局部的方向注入噪声，导致不必要的效用损失。为解决上述问题，我们提出选择性解耦联邦LoRA（SDFLoRA），这是一种结构感知的LoRA框架，将每个客户端更新解耦为用于聚合的共享组件和保留客户端特定语义的私有组件。仅共享组件参与子空间对齐，而私有组件保持本地且不参与通信，这使得训练与差分隐私兼容，并在秩异构环境下稳定聚合过程。通过仅向聚合后的共享更新注入噪声，该方法避免了对局部方向的扰动，提升了效用-隐私权衡。多基准实验表明，SDFLoRA优于联邦LoRA基线方法，并实现了出色的效用-隐私权衡。