Adapter tuning, which updates only a few parameters, has become a mainstream method for fine-tuning pretrained language models to downstream tasks. However, it often yields subpar results in few-shot learning. AdapterFusion, which assembles pretrained adapters using composition layers tailored to specific tasks, is a possible solution but significantly increases trainable parameters and deployment costs. Despite this, our preliminary study reveals that even single adapters can outperform Adapterfusion in few-shot learning, urging us to propose \textbf{\texttt{Merging Pretrained Adapters}} (MerA) that efficiently incorporates pretrained adapters to a single model through model fusion. Extensive experiments on two PLMs demonstrate that MerA achieves substantial improvements compared to both single adapters and AdapterFusion. To further enhance the capacity of MerA, we also introduce a simple yet effective technique, referred to as the "\textit{same-track}" setting, that merges adapters from the same track of pretraining tasks. With the implementation of the "\textit{same-track}" setting, we observe even more impressive gains, surpassing the performance of both full fine-tuning and adapter tuning by a substantial margin, e.g., 3.5\% in MRPC and 5.0\% in MNLI.

翻译：适配器调优仅需更新少量参数，已成为将预训练语言模型适配至下游任务的主流微调方法。然而，该方法在少样本学习场景中常表现欠佳。AdapterFusion通过为特定任务定制的组合层整合预训练适配器，虽为潜在解决方案，却显著增加了可训练参数与部署成本。我们的初步研究表明，即便是单一适配器在少样本学习中的表现也能超越AdapterFusion，这促使我们提出**预训练适配器融合方法**（MerA），通过模型融合将多个预训练适配器高效整合至单一模型。在两个预训练语言模型上的大量实验表明，MerA相较单一适配器与AdapterFusion均实现显著性能提升。为增强MerA的容量，我们进一步引入简单有效的"同路线"设置，将源自相同预训练任务路线的适配器进行融合。采用该设置后，我们观察到更显著的性能增益，在MRPC和MNLI任务上分别以3.5%与5.0%的绝对优势超越全参数微调与适配器调优。