Large models represent a groundbreaking advancement in multiple application fields, enabling remarkable achievements across various tasks. However, their unprecedented scale comes with significant computational costs. These models, often consisting of billions of parameters, require vast amounts of computational resources for execution. Especially, the expansive scale and computational demands pose considerable challenges when customizing them for particular downstream tasks, particularly over the hardware platforms constrained by computational capabilities. Parameter Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adjusting the large models over the various downstream tasks. In particular, PEFT refers to the process of adjusting the parameters of a pre-trained large models to adapt it to a specific task or domain while minimizing the number of additional parameters introduced or computational resources required. This approach is particularly important when dealing with large-scale language models with high parameter counts, as fine-tuning these models from scratch can be computationally expensive and resource-intensive, posing considerable challenges in the supporting system platform design. In this survey, we present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead. Moreover, we provide an overview of applications developed using different PEFT algorithms and discuss common techniques employed to mitigate computation costs for PEFT. In addition to providing an extensive survey from an algorithmic standpoint, we also examine various real-world system designs to investigate the implementation costs associated with different PEFT approaches. This survey serves as an indispensable resource for researchers aiming to understand both the PEFT algorithm and its system implementation, offering detailed ......

翻译：大型模型代表了多个应用领域的突破性进展，使得在各种任务中取得显著成就成为可能。然而，其前所未有的规模伴随着巨大的计算成本。这些通常包含数十亿参数的模型需要大量的计算资源来执行。特别是，当针对特定下游任务定制这些模型时，其庞大的规模与计算需求带来了相当大的挑战，尤其是在计算能力受限的硬件平台上。参数高效微调（PEFT）通过在各种下游任务上高效调整大型模型，提供了一种实用的解决方案。具体而言，PEFT指的是调整预训练大型模型的参数，使其适应特定任务或领域，同时最小化引入的额外参数数量或所需的计算资源。这种方法在处理参数量巨大的大规模语言模型时尤为重要，因为从头开始微调这些模型可能计算成本高昂且资源密集，给支撑系统平台设计带来相当大的挑战。在本综述中，我们对各种PEFT算法进行了全面研究，考察了其性能与计算开销。此外，我们概述了使用不同PEFT算法开发的应用，并讨论了用于降低PEFT计算成本的常用技术。除了从算法角度提供广泛综述外，我们还考察了各种现实世界的系统设计，以研究不同PEFT方法相关的实现成本。本综述旨在成为研究人员理解PEFT算法及其系统实现的不可或缺的资源，提供了详尽的……