Foundation Models (FMs), such as BERT, GPT, ViT, and CLIP, have demonstrated remarkable success in a wide range of applications, driven by their ability to leverage vast amounts of data for pre-training. However, optimizing FMs often requires access to sensitive data, raising privacy concerns and limiting their applicability in certain domains. In this paper, we introduce the concept of Federated Foundation Models (FFMs), a novel approach that combines the benefits of FMs and Federated Learning (FL) to enable privacy-preserving and collaborative learning across multiple institutions. We discuss the potential benefits and challenges of integrating FL into the lifespan of FMs, covering pre-training, fine-tuning, and application. We further provide formal definitions of FFM tasks, including FFM pre-training, FFM fine-tuning, and federated prompt engineering, allowing for more personalized and context-aware models while maintaining data privacy. Moreover, we explore the possibility of continual/lifelong learning in FFMs, as increased computational power at the edge unlocks the potential for optimizing FMs using newly generated private data at edges. We present experiments and evaluations comparing the performance of FFMs to traditional FMs on various downstream tasks, demonstrating the effectiveness of our approach in preserving privacy, reducing overfitting, and improving model generalizability. The proposed Federated Foundation Models offer a flexible and scalable framework for training large language models in a privacy-preserving manner, paving the way for future advancements in both FM pre-training and federated learning.

翻译：基础模型（Foundation Models, FMs），如BERT、GPT、ViT和CLIP，凭借其利用海量数据进行预训练的能力，已在广泛的应用中展现出卓越成效。然而，优化基础模型通常需要访问敏感数据，这引发了隐私担忧，并限制了其在特定领域的适用性。本文提出联邦基础模型（Federated Foundation Models, FFMs）这一概念，这是一种结合了基础模型与联邦学习（Federated Learning, FL）优势的新方法，旨在实现跨多个机构的隐私保护与协同学习。我们探讨了将联邦学习融入基础模型全生命周期（涵盖预训练、微调及应用）的潜在优势与挑战。我们进一步给出了联邦基础模型任务的正式定义，包括FFM预训练、FFM微调及联邦提示工程（federated prompt engineering），从而在维护数据隐私的同时，实现更具个性化和上下文感知的模型。此外，我们探索了FFM中持续/终身学习的可能性，因为边缘计算能力的提升为利用边缘新生成的私有数据优化基础模型释放了潜力。我们通过实验与评估，将FFM与传统FM在多种下游任务上的性能进行对比，证明了本方法在保护隐私、减少过拟合及提升模型泛化能力方面的有效性。所提出的联邦基础模型为以隐私保护方式训练大型语言模型提供了一个灵活且可扩展的框架，为基础模型预训练和联邦学习的未来发展铺平了道路。