Large models represent a groundbreaking advancement in multiple application fields, enabling remarkable achievements across various tasks. However, their unprecedented scale comes with significant computational costs. These models, often consisting of billions of parameters, require vast amounts of computational resources for execution. Especially, the expansive scale and computational demands pose considerable challenges when customizing them for particular downstream tasks, particularly over the hardware platforms constrained by computational capabilities. Parameter Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adapt the large models over the various downstream tasks. In particular, PEFT refers to the process of adjusting the parameters of a pre-trained large models to adapt it to a specific task while minimizing the number of additional parameters introduced or computational resources required. This approach is particularly important when dealing with large language models with high parameter counts, as fine-tuning these models from scratch can be computationally expensive and resource-intensive, posing considerable challenges in the supporting system platform design. In this survey, we present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead. Moreover, we provide an overview of applications developed using different PEFT algorithms and discuss common techniques employed to mitigate computation costs for PEFT. In addition to the algorithmic perspective, we overview various real-world system designs to investigate the implementation costs associated with different PEFT algorithms. This survey serves as an indispensable resource for researchers aiming to understand both the PEFT algorithm and its system implementation, offering detailed insights into recent advancements and practical applications.

翻译：大型模型在多个应用领域代表了一项突破性进展，能够在各类任务中取得显著成果。然而，其前所未有的规模带来了巨大的计算成本。这些模型通常包含数十亿个参数，在执行时需要大量计算资源。特别是，当将其定制到特定下游任务时，尤其是在计算能力受限的硬件平台上，其庞大的规模和计算需求构成了相当大的挑战。参数高效微调（Parameter Efficient Fine-Tuning, PEFT）提供了一种实用解决方案，能够高效地使大型模型适应各种下游任务。具体而言，PEFT指的是调整预训练大型模型参数以使其适配特定任务的过程，同时最大限度地减少引入的额外参数数量或所需计算资源。这种方法在处理具有高参数数量的大型语言模型时尤为重要，因为从头开始微调这些模型在计算上可能代价高昂且资源密集，对支持系统平台的设计带来重大挑战。在本综述中，我们全面研究了各种PEFT算法，审视了它们的性能和计算开销。此外，我们概述了使用不同PEFT算法开发的应用，并讨论了为降低PEFT计算成本而常用的技术。除了算法层面，我们还综述了各种实际系统设计，以探究不同PEFT算法的实现成本。本综述旨在为希望理解PEFT算法及其系统实现的研究人员提供不可或缺的资源，并详细阐述最新进展与实际应用。