Federated learning (FL) for large language models (LLMs) has attracted increasing attention as a way to enable privacy-preserving adaptation over distributed data. Parameter-efficient methods such as LoRA are widely adopted to reduce communication and memory costs. Despite these advances, practical FL deployments often exhibit rank heterogeneity, since different clients may use different low-rank configurations. This makes direct aggregation of LoRA updates biased and unstable. Existing solutions typically enforce unified ranks or align heterogeneous updates into a shared subspace, which over-constrains client-specific semantics, limits personalization, and provides weak protection of local client information under differential privacy noise. To address this issue, we propose Selective Dual-module Federated LoRA (SDFLoRA), which decomposes each client adapter into a global module that captures transferable knowledge and a local module that preserves client-specific adaptations. The global module is selectively aligned and aggregated across clients, while local modules remain private. This design enables robust learning under rank heterogeneity and supports privacy-aware optimization by injecting differential privacy noise exclusively into the global module. Experiments on GLUE benchmarks demonstrate that SDFLoRA outperforms representative federated LoRA baselines and achieves a better utility-privacy trade-off.

翻译：大语言模型（LLM）的联邦学习（FL）作为一种能够在分布式数据上实现隐私保护式适配的方法，正受到越来越多的关注。诸如LoRA等参数高效方法被广泛采用以降低通信与内存开销。尽管取得了这些进展，但实际的FL部署常常表现出秩异构性，因为不同的客户端可能采用不同的低秩配置。这使得直接聚合LoRA更新会产生偏差且不稳定。现有解决方案通常强制统一秩或将异构更新对齐到共享子空间，这过度约束了客户端特定的语义，限制了个性化能力，并且在差分隐私噪声下对本地客户端信息的保护较弱。为解决此问题，我们提出了选择性双模块联邦LoRA（SDFLoRA），它将每个客户端适配器分解为一个捕获可迁移知识的全局模块和一个保留客户端特定适配的本地模块。全局模块在客户端间进行选择性对齐与聚合，而本地模块则保持私有。这种设计使得在秩异构条件下能够进行鲁棒学习，并通过将差分隐私噪声仅注入全局模块来支持隐私感知优化。在GLUE基准测试上的实验表明，SDFLoRA优于代表性的联邦LoRA基线方法，并实现了更好的效用-隐私权衡。