Parameter-efficient fine-tuning (PEFT) techniques, such as adapter tuning, aim to fine-tune a pre-trained language model (PLM) using a minimal number of parameters for a specific task or profile. Although adapter tuning provides increased parameter efficiency compared to full-model fine-tuning, it introduces a small set of additional parameters attached to a PLM for each profile. This can become problematic in practical applications with multiple profiles, particularly when a significant increase in the number of profiles linearly boosts the total number of additional parameters. To mitigate this issue, we introduce X-PEFT, a novel PEFT method that leverages a multitude of given adapters by fine-tuning an extremely small set of compact tensors for a new profile, which serve as binary masks to adaptively select the given adapters. To efficiently validate our proposed method, we implement it using a large number of trained or untrained (random) adapters. We evaluate the performance of X-PEFT through LaMP and GLUE tasks and demonstrate that it either matches or surpasses the effectiveness of conventional adapter tuning, despite reducing the memory requirements per profile by a factor of 10,000 compared to it.

翻译：参数高效微调（PEFT）技术（如适配器调优）旨在通过最少参数针对特定任务或配置文件对预训练语言模型（PLM）进行微调。尽管适配器调优相比全模型微调提升了参数效率，但它仍为每个配置文件在PLM上引入少量附加参数。在实际多配置文件应用中，当配置文件数量线性增长导致附加参数总量显著增加时，这一问题变得尤为突出。为解决此问题，我们提出X-PEFT——一种新型PEFT方法，通过为新配置文件微调一组极其紧凑的张量（作为二元掩码自适应选择已有适配器）来利用大量给定的适配器。为高效验证所提方法，我们利用大量已训练或未训练（随机）适配器实现该方案。通过LaMP和GLUE任务的评估表明，尽管每个配置文件的存储需求相较于传统适配器调优降低了10,000倍，X-PEFT仍能在效率上匹敌甚至超越传统适配器调优方法的有效性。