Large models represent a groundbreaking advancement in multiple application fields, enabling remarkable achievements across various tasks. However, their unprecedented scale comes with significant computational costs. These models, often consisting of billions of parameters, require vast amounts of computational resources for execution. Especially, the expansive scale and computational demands pose considerable challenges when customizing them for particular downstream tasks, particularly over the hardware platforms constrained by computational capabilities. Parameter Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adapt the large models over the various downstream tasks. In particular, PEFT refers to the process of adjusting the parameters of a pre-trained large models to adapt it to a specific task while minimizing the number of additional parameters introduced or computational resources required. This approach is particularly important when dealing with large language models with high parameter counts, as fine-tuning these models from scratch can be computationally expensive and resource-intensive, posing considerable challenges in the supporting system platform design. In this survey, we present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead. Moreover, we provide an overview of applications developed using different PEFT algorithms and discuss common techniques employed to mitigate computation costs for PEFT. In addition to the algorithmic perspective, we overview various real-world system designs to investigate the implementation costs associated with different PEFT algorithms. This survey serves as an indispensable resource for researchers aiming to understand both the PEFT algorithm and its system implementation, offering detailed insights into recent advancements and practical applications.

翻译：大型模型在多个应用领域代表了突破性进展，实现了各类任务的卓越成果。然而，其前所未有的规模带来了显著的计算成本。这些模型通常包含数十亿参数，需要大量计算资源才能运行。尤其在针对特定下游任务进行定制化调整时，其庞大的规模和计算需求对计算能力受限的硬件平台构成了巨大挑战。参数高效微调（Parameter Efficient Fine-Tuning, PEFT）提供了一种实用解决方案，能够高效地使大型模型适应各种下游任务。具体而言，PEFT是指在最小化引入的额外参数数量或所需计算资源的前提下，调整预训练大型模型的参数以适应特定任务的过程。这种方法在处理具有高参数数量的大型语言模型时尤为重要，因为从头微调这些模型可能计算成本高昂且资源密集，给支持系统平台设计带来巨大挑战。本综述全面研究了各类PEFT算法，评估了它们的性能与计算开销。此外，我们概述了基于不同PEFT算法开发的应用，并讨论了常用于降低PEFT计算成本的通用技术。除算法视角外，我们还考察了多种实际系统设计，以探究不同PEFT算法的实现成本。本综述致力于成为研究人员理解PEFT算法及其系统实现的不可或缺资源，提供了关于最新进展与实用应用的详细见解。