Foundation Models (FMs), such as LLaMA, BERT, GPT, ViT, and CLIP, have demonstrated remarkable success in a wide range of applications, driven by their ability to leverage vast amounts of data for pre-training. However, optimizing FMs often requires access to sensitive data, raising privacy concerns and limiting their applicability in many domains. In this paper, we propose the Federated Foundation Models (FFMs) paradigm, which combines the benefits of FMs and Federated Learning (FL) to enable privacy-preserving and collaborative learning across multiple end-users. We discuss the potential benefits and challenges of integrating FL into the lifespan of FMs, covering pre-training, fine-tuning, and application. We further outline potential future research avenues in FFM, including FFM pre-training, FFM fine-tuning, and federated prompt tuning, which allow the development of more personalized and context-aware models while ensuring data privacy. Moreover, we explore the possibility of continual/lifelong learning in FFMs, as increased computational power at the edge may unlock the potential for optimizing FMs using newly generated private data close to the data source. The proposed FFM concepts offer a flexible and scalable framework for training large language models in a privacy-preserving manner, setting the stage for subsequent advancements in both FM training and federated learning.

翻译：基础模型（如LLaMA、BERT、GPT、ViT和CLIP）凭借其利用海量数据进行预训练的能力，已在广泛的应用场景中展现出卓越成效。然而，优化基础模型常常需要访问敏感数据，这引发了隐私担忧并限制了其在许多领域的适用性。本文提出联邦基础模型范式，该范式融合了基础模型与联邦学习的优势，能够在多个终端用户之间实现隐私保护下的协作学习。我们探讨了将联邦学习融入基础模型全生命周期（涵盖预训练、微调及应用阶段）的潜在收益与挑战，并进一步勾勒了联邦基础模型中未来可能的研究方向，包括联邦基础模型预训练、微调以及联邦提示调优等，这些方向能够在保障数据隐私的前提下开发出更具个性化和上下文感知能力的模型。此外，本文探索了联邦基础模型中持续/终身学习的可能性——随着边缘端计算能力的增强，利用靠近数据源的新生成私有数据优化基础模型的潜力有望得到释放。所提出的联邦基础模型概念为以隐私保护方式训练大型语言模型提供了灵活且可扩展的框架，为基础模型训练与联邦学习的后续发展奠定了坚实基础。