Federated Learning (FL) offers a decentralized framework for training and fine-tuning Large Language Models (LLMs) by leveraging computational resources across organizations while keeping sensitive data on local devices. It addresses privacy and security concerns while navigating challenges associated with the substantial computational demands of LLMs, which can be prohibitive for small and medium-sized organizations. FL supports the development of task-specific LLMs for cross-silo applications through fine-tuning but remains vulnerable to inference-related risks that threaten sensitive information. Prior studies have utilized Differential Privacy (DP) in LLM fine-tuning, which, despite being effective at preserving privacy, can degrade model performance. To overcome these challenges, we propose FedShield-LLM which integrates pruning with Fully Homomorphic Encryption (FHE) applied to Low-Rank Adaptation (LoRA) parameters. This combination enables secure computation over encrypted model updates and reduces the attack surface by deactivating less important LoRA parameters. Furthermore, optimized federated algorithms for cross-silo environments enhance scalability and efficiency. Parameter-efficient fine-tuning techniques like LoRA substantially reduce computational and communication overhead, making FL feasible for resource-constrained clients. Extensive experiments using Llama-2 models (7B and 13B) on four diverse datasets demonstrate that FedShield-LLM achieves superior collaborative performance and system efficiency compared to existing methods, supporting practical deployment across multiple domains.

翻译：联邦学习（FL）通过跨组织利用计算资源、同时将敏感数据保留在本地设备上，为训练和微调大语言模型（LLM）提供了去中心化框架。它在解决隐私和安全问题的同时，应对了与LLM巨大计算需求相关的挑战——这种需求对中小型组织而言可能难以承受。FL支持通过微调开发面向跨孤岛应用的任务特定LLM，但仍易受威胁敏感信息的推理相关风险影响。先前研究在LLM微调中采用差分隐私（DP），但该方法虽能有效保护隐私，却可能降低模型性能。为克服这些挑战，我们提出了FedShield-LLM，该模型将剪枝与应用于低秩适应（LoRA）参数的全同态加密（FHE）相结合。这种组合方式能在加密模型更新上实现安全计算，并通过停用重要性较低的LoRA参数来减少攻击面。此外，针对跨孤岛环境优化的联邦算法增强了可扩展性和效率。LoRA等参数高效微调技术显著降低了计算和通信开销，使FL对资源受限客户端具有可行性。基于Llama-2模型（7B和13B）在四个不同数据集上的大量实验表明，与现有方法相比，FedShield-LLM实现了更优越的协作性能与系统效率，支持在多个领域的实际部署。