Large pretrained language models have been widely used in downstream NLP tasks via task-specific fine-tuning. Recently, an array of Parameter-Efficient Fine-Tuning (PEFT) methods have also achieved strong task performance while updating a much smaller number of parameters compared to full model tuning. However, it is non-trivial to make informed per-task design choices (i.e., to create PEFT configurations) concerning the selection of PEFT architectures and modules, the number of tunable parameters, and even the layers in which the PEFT modules are inserted. Consequently, it is highly likely that the current, manually set PEFT configurations might be suboptimal for many tasks from the perspective of the performance-to-efficiency trade-off. To address the core question of the PEFT configuration selection that aims to control and maximise the balance between performance and parameter efficiency, we first define a rich configuration search space spanning multiple representative PEFT modules along with finer-grained configuration decisions over the modules (e.g., parameter budget, insertion layer). We then propose AutoPEFT, a novel framework to traverse this configuration space: it automatically configures multiple PEFT modules via high-dimensional Bayesian optimisation. We show the resource scalability and task transferability of AutoPEFT-found configurations, outperforming existing PEFT methods on average on the standard GLUE benchmark while conducting the configuration search on a single task. The per-task AutoPEFT-based configuration search even outperforms full-model fine-tuning.

翻译：大型预训练语言模型已通过任务特定的微调广泛应用于下游自然语言处理任务中。近年来，一系列参数高效微调（PEFT）方法在仅更新远少于全模型微调参数数量的同时，也取得了强大的任务性能。然而，在涉及PEFT架构与模块的选择、可调参数数量乃至PEFT模块插入层等设计决策时（即构建PEFT配置），要做出明智的逐任务选择并非易事。因此，从性能与效率权衡的角度来看，当前手动设置的PEFT配置很可能对许多任务而言并非最优。为了应对PEFT配置选择的核心问题——该问题旨在控制并最大化性能与参数效率之间的平衡——我们首先定义了一个丰富的配置搜索空间，该空间涵盖多种代表性PEFT模块及其更细粒度的配置决策（例如参数预算、插入层）。随后，我们提出AutoPEFT——一种全新的框架以遍历该配置空间：它通过高维贝叶斯优化自动配置多种PEFT模块。我们展示了AutoPEFT所发现配置的资源可扩展性与任务可迁移性：在标准GLUE基准测试上，尽管配置搜索仅在单个任务上进行，其平均性能仍优于现有PEFT方法。基于AutoPEFT的逐任务配置搜索甚至超越了全模型微调。