Foundation Models (FMs), such as LLaMA, BERT, GPT, ViT, and CLIP, have demonstrated remarkable success in a wide range of applications, driven by their ability to leverage vast amounts of data for pre-training. However, optimizing FMs often requires access to sensitive data, raising privacy concerns and limiting their applicability in many domains. In this paper, we propose the Federated Foundation Models (FFMs) paradigm, which combines the benefits of FMs and Federated Learning (FL) to enable privacy-preserving and collaborative learning across multiple end-users. We discuss the potential benefits and challenges of integrating FL into the lifespan of FMs, covering pre-training, fine-tuning, and application. We further outline potential future research avenues in FFM, including FFM pre-training, FFM fine-tuning, and federated prompt tuning, which allow the development of more personalized and context-aware models while ensuring data privacy. Moreover, we explore the possibility of continual/lifelong learning in FFMs, as increased computational power at the edge may unlock the potential for optimizing FMs using newly generated private data close to the data source. The proposed FFM concepts offer a flexible and scalable framework for training large language models in a privacy-preserving manner, setting the stage for subsequent advancements in both FM training and federated learning.

翻译：基础模型（如LLaMA、BERT、GPT、ViT和CLIP）凭借其利用海量数据进行预训练的能力，在众多应用中取得了显著成功。然而，优化这些模型通常需要访问敏感数据，由此引发隐私问题并限制了其在许多领域的适用性。本文提出联邦基础模型（FFM）范式，该范式融合了基础模型与联邦学习（FL）的优势，能够在多个终端用户之间实现隐私保护与协同学习。我们探讨了将联邦学习融入基础模型全生命周期（涵盖预训练、微调及应用）的潜在优势与挑战。进一步阐述了FFM领域未来可能的研究方向，包括FFM预训练、FFM微调及联邦提示调优，这些方法可在保障数据隐私的前提下，开发出更具个性化和上下文感知能力的模型。此外，我们探讨了FFM中持续/终身学习的可能性——随着边缘计算能力的提升，利用数据源附近新生成的私有数据优化基础模型的潜力或将得以释放。所提出的FFM概念为以隐私保护方式训练大型语言模型提供了灵活且可扩展的框架，为基础模型训练与联邦学习的后续发展奠定了基础。