Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. However, new approaches to FL often discuss their contributions involving small deep-learning models only and focus on training full models on clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. Typically, FMs have already been pre-trained across a wide variety of tasks and can be fine-tuned to specific downstream tasks over significantly smaller datasets than required for full model training. However, access to such datasets is often challenging. By its design, FL can help to open data silos. With this survey, we introduce a novel taxonomy focused on computational and communication efficiency, the vital elements to make use of FMs in FL systems. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications, elaborate on the readiness of FL frameworks to work with FMs and provide future research opportunities on how to evaluate generative models in FL as well as the interplay of privacy and PEFT.

翻译：联邦学习（FL）已成为一种成熟技术，旨在促进跨多个客户端的隐私保护协同训练。然而，新的FL方法通常仅针对小型深度学习模型讨论其贡献，并侧重于在客户端上训练完整模型。随着基础模型（FM）的出现，许多深度学习应用的实际情况已发生改变。通常情况下，基础模型已通过广泛任务完成预训练，可通过远少于全模型训练所需的数据集，针对特定下游任务进行微调。然而，获取此类数据集往往极具挑战。从设计上看，联邦学习能助力打破数据孤岛。通过本综述，我们提出了一种聚焦于计算效率与通信效率的新型分类体系——这是联邦学习系统中有效利用基础模型的关键要素。我们探讨了参数高效微调（PEFT）在联邦学习应用中的利弊，阐释了联邦学习框架与基础模型协同工作的就绪程度，并提出了评估联邦学习中生成式模型、以及隐私保护与参数高效微调相互作用等未来研究方向。