Recent parameter-efficient finetuning (PEFT) techniques aim to improve over the considerable cost of fully finetuning large pretrained language models (PLM). As different PEFT techniques proliferate, it is becoming difficult to compare them, in particular in terms of (i) the structure and functionality they add to the PLM, (ii) the different types and degrees of efficiency improvements achieved, (iii) performance at different downstream tasks, and (iv) how differences in structure and functionality relate to efficiency and task performance. To facilitate such comparisons, this paper presents a reference framework which standardises aspects shared by different PEFT techniques, while isolating differences to specific locations and interactions with the standard components. Through this process of standardising and isolating differences, a modular view of PEFT techniques emerges, supporting not only direct comparison of different techniques and their efficiency and task performance, but also systematic exploration of reusability and composability of the different types of finetuned modules. We demonstrate how the reference framework can be applied to understand properties and relative advantages of PEFT techniques, hence to inform selection of techniques for specific tasks, and design choices for new PEFT techniques.

翻译：近期参数高效微调（PEFT）技术旨在改进大型预训练语言模型（PLM）全参数微调的高昂成本。随着不同PEFT技术的涌现，对其进行有效比较变得日益困难，尤其体现在：（i）它们为PLM增添的结构与功能特性；（ii）实现的效率改进类型与程度差异；（iii）不同下游任务上的性能表现；（iv）结构功能差异如何影响效率与任务性能。为促进此类比较，本文提出一个参考框架，该框架统一不同PEFT技术的共享特性，同时将差异定位至特定模块及其与标准组件的交互关系。通过这种标准化与差异隔离的过程，PEFT技术呈现出模块化视图，不仅支持直接比较不同技术的效率与任务性能，还可系统探索不同类型微调模块的可复用性与可组合性。我们展示了如何应用该参考框架理解PEFT技术的特性与相对优势，从而为特定任务的技术选择提供依据，并为新型PEFT技术的设计决策提供指导。