Adapter-based tuning has recently arisen as an alternative to fine-tuning. It works by adding light-weight adapter modules to a pretrained language model (PrLM) and only updating the parameters of adapter modules when learning on a downstream task. As such, it adds only a few trainable parameters per new task, allowing a high degree of parameter sharing. Prior studies have shown that adapter-based tuning often achieves comparable results to fine-tuning. However, existing work only focuses on the parameter-efficient aspect of adapter-based tuning while lacking further investigation on its effectiveness. In this paper, we study the latter. We first show that adapter-based tuning better mitigates forgetting issues than fine-tuning since it yields representations with less deviation from those generated by the initial PrLM. We then empirically compare the two tuning methods on several downstream NLP tasks and settings. We demonstrate that 1) adapter-based tuning outperforms fine-tuning on low-resource and cross-lingual tasks; 2) it is more robust to overfitting and less sensitive to changes in learning rates.

翻译：最近出现了适应器调试,作为微调的替代方法。它的工作是将轻量适配器模块添加到预先培训的语言模型(PrLM)中,并且只在学习下游任务时更新适应器模块的参数。因此,它只为每个新任务增加几个可培训的参数,允许高度的参数共享。以前的研究表明,基于适应器的调试往往取得与微调相类似的效果。然而,现有工作只侧重于基于适应器的调试的参数效率方面,而没有对其有效性进行进一步调查。在本文中,我们研究了后者。我们首先表明,基于适应器的调试比微调更能减轻忘记的问题,因为它产生与最初的PrLM产生的调试样的偏差较少。我们随后对下游国家调整器的任务和设置的两个调控方法进行了实验性比较。我们证明,1)基于适应器的调试在低资源和跨语言任务上进行微调试;2)它更强有力地适应和不太敏感地注意学习率的变化。