With the continuous growth in the number of parameters of transformer-based pretrained language models (PLMs), particularly the emergence of large language models (LLMs) with billions of parameters, many natural language processing (NLP) tasks have demonstrated remarkable success. However, the enormous size and computational demands of these models pose significant challenges for adapting them to specific downstream tasks, especially in environments with limited computational resources. Parameter Efficient Fine-Tuning (PEFT) offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. The demands for fine-tuning PLMs, especially LLMs, have led to a surge in the development of PEFT methods, as depicted in Fig. 1. In this paper, we present a comprehensive and systematic review of PEFT methods for PLMs. We summarize these PEFT methods, discuss their applications, and outline future directions. Furthermore, we conduct experiments using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency. By offering insights into the latest advancements and practical applications, this survey serves as an invaluable resource for researchers and practitioners seeking to navigate the challenges and opportunities presented by PEFT in the context of PLMs.

翻译：随着基于Transformer的预训练语言模型（PLMs）参数规模持续增长，特别是数十亿参数级别的大语言模型（LLMs）的出现，众多自然语言处理（NLP）任务取得了显著成功。然而，这些模型庞大的规模与计算需求对将其适配到特定下游任务构成了重大挑战，尤其是在计算资源有限的环境下。参数高效微调（PEFT）通过减少微调参数数量和内存占用，在达到与全量微调相当性能的同时，提供了一种有效的解决方案。微调PLMs（特别是LLMs）的需求推动了PEFT方法的迅猛发展，如图1所示。本文对PLMs的PEFT方法进行了全面且系统的评述。我们总结了这些PEFT方法，探讨了其应用场景，并展望了未来发展方向。此外，我们采用若干代表性PEFT方法开展实验，以更深入地理解其在参数效率与内存效率方面的有效性。通过揭示最新进展与实践应用，本综述为面临PLMs背景下PEFT带来的机遇与挑战的研究人员和从业者提供了宝贵的参考资源。