Low-rank adaptation (LoRA) has emerged as a powerful tool for parameter-efficient fine-tuning of large language models (LLMs). This paper studies LoRA under a federated learning setting, enabling collaborative fine-tuning across clients while preserving parameter efficiency. We focus on a highly heterogeneous regime in which clients share only partial structure and a substantial subset may be contaminated. We propose Collaborative Low-rank Alignment and Identifiable Recovery (CLAIR), a contamination-aware framework that relies only on preliminary local estimators. Its formulation applies broadly, from linear regression to neural network and LLM modules, whenever local adaptation can be represented by matrix-valued updates. CLAIR recovers the shared LoRA subspace and detects contaminated clients via a structured low-rank plus block-sparse decomposition. We prove exact recovery of the shared LoRA subspace in the noiseless case, stable recovery under preliminary estimation error, and consistent collaborative-set recovery under mild separation conditions. We further quantify the gain from CLAIR refinement: it reduces off-subspace estimation error through cross-client averaging while preserving client-specific variation within the shared LoRA subspace, thus improves over local fine-tuning whenever this oracle gain outweighs the costs of subspace estimation and benign-client heterogeneity. Empirically, we demonstrate the benefits of CLAIR by fine-tuning a Transformer architecture on a text-copying task. The results show accurate contamination detection and improved benign-client performance compared with local fine-tuning and non-robust federated averaging.

翻译：低秩适配（LoRA）已成为大语言模型参数高效微调的有力工具。本文研究联邦学习场景下的LoRA方法，在保持参数效率的同时实现跨客户端的协作微调。我们聚焦于高度异构的场景：客户端仅共享部分结构，且存在大量污染客户端。本文提出协作低秩对齐与可辨识恢复（CLAIR）框架，该框架仅依赖初步局部估计量，具备抗污染能力。其公式具有广泛适用性——从线性回归到神经网络及大语言模型模块，只要局部适配可表示为矩阵值更新。CLAIR通过结构化低秩加块稀疏分解，在共享LoRA子空间恢复的同时检测污染客户端。我们证明了无噪声场景下共享LoRA子空间的精确恢复、初步估计误差下的稳定恢复，以及温和分离条件下的协作集一致性恢复。进一步量化了CLAIR精炼的增益：通过跨客户端平均降低子空间外估计误差，同时保留共享LoRA子空间内的客户端特异性变异——当这种理论增益超过子空间估计与良性客户端异质性的成本时，即优于局部微调。在实证方面，我们通过在文本复制任务上微调Transformer架构验证了CLAIR的优势。结果表明，与局部微调及非鲁棒联邦平均相比，本方法实现了精确的污染检测与提升的良性客户端性能。