Large scale language models (LLM) have received significant attention and found diverse applications across various domains, but their development encounters challenges in real-world scenarios. These challenges arise due to the scarcity of public domain data availability and the need to maintain privacy with respect to private domain data. To address these issues, federated learning (FL) has emerged as a promising technology that enables collaborative training of shared models while preserving decentralized data. We propose the concept of federated LLM, which comprises three key components, i.e., federated LLM pre-training, federated LLM fine-tuning, and federated LLM prompt engineering. For each component, we discuss its advantage over traditional LLM training methods and propose specific engineering strategies for implementation. Furthermore, we explore the novel challenges introduced by the integration of FL and LLM. We analyze existing solutions and identify potential obstacles faced by these solutions within the context of federated LLM.

翻译：大规模语言模型（LLM）已受到广泛关注，并在多个领域展现出多样化应用，但其在现实场景中的开发面临诸多挑战。这些挑战源于公共领域数据的稀缺性以及需维护私有领域数据的隐私性。为解决上述问题，联邦学习（FL）作为一种能够实现共享模型协同训练且保持数据去中心化的新兴技术应运而生。我们提出联邦大语言模型（Federated LLM）的概念，该概念包含三个关键组成部分，即联邦LLM预训练、联邦LLM微调以及联邦LLM提示工程。针对每个组成部分，我们探讨其相较于传统LLM训练方法的优势，并提出具体的工程实施策略。此外，我们进一步探究联邦学习与LLM融合所引入的新挑战，分析现有解决方案，并识别这些方案在联邦LLM场景下所面临的潜在障碍。