Fine-tuning pre-trained language models for multiple tasks tends to be expensive in terms of storage. To mitigate this, parameter-efficient transfer learning (PETL) methods have been proposed to address this issue, but they still require a significant number of parameters and storage when being applied to broader ranges of tasks. To achieve even greater storage reduction, we propose PROPETL, a novel method that enables efficient sharing of a single PETL module which we call prototype network (e.g., adapter, LoRA, and prefix-tuning) across layers and tasks. We then learn binary masks to select different sub-networks from the shared prototype network and apply them as PETL modules into different layers. We find that the binary masks can determine crucial information from the network, which is often ignored in previous studies. Our work can also be seen as a type of pruning method, where we find that overparameterization also exists in the seemingly small PETL modules. We evaluate PROPETL on various downstream tasks and show that it can outperform other PETL methods with approximately 10% of the parameter storage required by the latter.

翻译：针对多任务微调预训练语言模型在存储方面成本高昂的问题，参数高效迁移学习方法被提出以缓解此困境。然而，当应用于更广泛的任务范围时，这些方法仍需要大量参数和存储空间。为实现更大程度的存储缩减，我们提出PROPETL——一种新颖方法，它能够跨层跨任务高效共享单个PETL模块（我们称之为原型网络，例如适配器、LoRA和前缀微调）。随后，我们学习二进制掩码，从共享的原型网络中选取不同的子网络，并将其作为PETL模块应用于不同网络层。我们发现，二进制掩码能提取网络中关键信息，这一特性在以往研究中常被忽略。我们的工作也可视为一种剪枝方法，揭示了即便在看似微小的PETL模块中也存在过参数化现象。我们在多种下游任务上评估PROPETL，结果表明其仅需其他PETL方法约10%的参数存储量即可超越后者的性能。