Large models represent a groundbreaking advancement in multiple application fields, enabling remarkable achievements across various tasks. However, their unprecedented scale comes with significant computational costs. These models, often consisting of billions of parameters, require vast amounts of computational resources for execution. Especially, the expansive scale and computational demands pose considerable challenges when customizing them for particular downstream tasks, particularly over the hardware platforms constrained by computational capabilities. Parameter Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adapt the large models over the various downstream tasks. In particular, PEFT refers to the process of adjusting the parameters of a pre-trained large models to adapt it to a specific task while minimizing the number of additional parameters introduced or computational resources required. This approach is particularly important when dealing with large language models with high parameter counts, as fine-tuning these models from scratch can be computationally expensive and resource-intensive, posing considerable challenges in the supporting system platform design. In this survey, we present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead. Moreover, we provide an overview of applications developed using different PEFT algorithms and discuss common techniques employed to mitigate computation costs for PEFT. In addition to the algorithmic perspective, we overview various real-world system designs to investigate the implementation costs associated with different PEFT algorithms. This survey serves as an indispensable resource for researchers aiming to understand both the PEFT algorithm and its system implementation, offering detailed insights into recent advancements and practical applications.

翻译：大模型代表了多个应用领域的突破性进展，在各种任务中取得了显著成就。然而，其前所未有的规模带来了高昂的计算成本。这些模型通常包含数十亿参数，需要海量计算资源才能运行。特别是，当针对特定下游任务进行定制时，尤其是在计算能力受限的硬件平台上，其庞大的规模和计算需求带来了巨大挑战。参数高效微调（PEFT）提供了一种实用解决方案，通过高效地使大模型适应各种下游任务。具体而言，PEFT指的是调整预训练大模型参数以使其适应特定任务的过程，同时最小化引入的额外参数数量或所需计算资源。在处理具有高参数数量的大语言模型时，这种方法尤为重要，因为从头微调这些模型在计算上可能非常昂贵且资源密集，对支撑系统平台设计构成重大挑战。在本综述中，我们全面研究了多种PEFT算法，考察了它们的性能和计算开销。此外，我们概述了使用不同PEFT算法开发的应用，并讨论了常用于降低PEFT计算成本的技术。除了算法视角，我们还概述了多种实际系统设计，以研究不同PEFT算法相关的实现成本。本综述旨在成为研究人员理解PEFT算法及其系统实现不可或缺的资源，提供关于最新进展和实际应用的详细见解。