With increasing privacy concerns on data, recent studies have made significant progress using federated learning (FL) on privacy-sensitive natural language processing (NLP) tasks. Much literature suggests fully fine-tuning pre-trained language models (PLMs) in the FL paradigm can mitigate the data heterogeneity problem and close the performance gap with centralized training. However, large PLMs bring the curse of prohibitive communication overhead and local model adaptation costs for the FL system. To this end, we introduce various parameter-efficient tuning (PETuning) methods into federated learning. Specifically, we provide a holistic empirical study of representative PLMs tuning methods in FL. The experimental results cover the analysis of data heterogeneity levels, data scales, and different FL scenarios. Overall communication overhead can be significantly reduced by locally tuning and globally aggregating lightweight model parameters while maintaining acceptable performance in various FL settings. To facilitate the research of PETuning in FL, we also develop a federated tuning framework FedPETuning, which allows practitioners to exploit different PETuning methods under the FL training paradigm conveniently. The source code is available at \url{https://github.com/iezhuozhuo/FedETuning/tree/deltaTuning}.

翻译：随着对数据隐私日益增长的关注，近期研究在隐私敏感的自然语言处理任务中利用联邦学习取得了显著进展。大量文献表明，在联邦学习范式下对预训练语言模型进行全参数微调能够缓解数据异构问题，并缩小与集中式训练的性能差距。然而，大型预训练语言模型给联邦系统带来了通信开销和本地模型适配成本过高的难题。为此，我们将多种参数高效微调方法引入联邦学习。具体而言，我们对联邦学习中具有代表性的预训练语言模型微调方法进行了全面的实证研究。实验结果涵盖了数据异构程度、数据规模以及不同联邦学习场景的分析。通过在本地微调并全局聚合轻量级模型参数，整体通信开销得以显著降低，同时在各种联邦学习设置下仍能保持可接受的性能。为促进联邦学习中参数高效微调的研究，我们还开发了一个联邦微调框架FedPETuning，使得研究者能够便捷地在联邦学习训练范式下利用不同的参数高效微调方法。源代码见\url{https://github.com/iezhuozhuo/FedETuning/tree/deltaTuning}。