Pre-trained language models (PLM) have revolutionized the NLP landscape, achieving stellar performances across diverse tasks. These models, while benefiting from vast training data, often require fine-tuning on specific data to cater to distinct downstream tasks. However, this data adaptation process has inherent security and privacy concerns, primarily when leveraging user-generated, device-residing data. Federated learning (FL) provides a solution, allowing collaborative model fine-tuning without centralized data collection. However, applying FL to finetune PLMs is hampered by challenges, including restricted model parameter access, high computational requirements, and communication overheads. This paper introduces Federated Black-box Prompt Tuning (FedBPT), a framework designed to address these challenges. FedBPT does not require the clients to access the model parameters. By focusing on training optimal prompts and utilizing gradient-free optimization methods, FedBPT reduces the number of exchanged variables, boosts communication efficiency, and minimizes computational and storage costs. Experiments highlight the framework's ability to drastically cut communication and memory costs while maintaining competitive performance. Ultimately, FedBPT presents a promising solution for efficient, privacy-preserving fine-tuning of PLM in the age of large language models.

翻译：预训练语言模型（PLM）已彻底改变自然语言处理领域，在各类任务中均取得卓越性能。这类模型受益于海量训练数据，但通常需要针对特定数据进行微调以适应不同的下游任务。然而，这一数据适应过程存在固有的安全与隐私问题，尤其在利用用户生成、驻留设备的数据时更为突出。联邦学习（FL）提供了一种解决方案，可在无需集中收集数据的情况下实现协同模型微调。但将FL应用于PLM微调仍面临诸多挑战，包括受限的模型参数访问、高计算需求以及通信开销。本文提出联邦黑盒提示调优（FedBPT）框架，旨在解决上述挑战。FedBPT无需客户端访问模型参数，通过聚焦于最优提示训练并采用无梯度优化方法，减少了交换变量数量，提升了通信效率，同时降低了计算和存储成本。实验结果表明，该框架在保持竞争性性能的同时，能显著降低通信与内存开销。最终，FedBPT为大语言模型时代高效、隐私保护的PLM微调提供了一种有前景的解决方案。