Parameter-efficient transfer learning (PETL), i.e., fine-tuning a small portion of parameters, is an effective strategy for adapting pre-trained models to downstream domains. To further reduce the memory demand, recent PETL works focus on the more valuable memory-efficient characteristic. In this paper, we argue that the scalability, adaptability, and generalizability of state-of-the-art methods are hindered by structural dependency and pertinency on specific pre-trained backbones. To this end, we propose a new memory-efficient PETL strategy, Universal Parallel Tuning (UniPT), to mitigate these weaknesses. Specifically, we facilitate the transfer process via a lightweight and learnable parallel network, which consists of: 1) A parallel interaction module that decouples the sequential connections and processes the intermediate activations detachedly from the pre-trained network. 2) A confidence aggregation module that learns optimal strategies adaptively for integrating cross-layer features. We evaluate UniPT with different backbones (e.g., T5, VSE$\infty$, CLIP4Clip, Clip-ViL, and MDETR) on various vision-and-language and pure NLP tasks. Extensive ablations on 18 datasets have validated that UniPT can not only dramatically reduce memory consumption and outperform the best competitor, but also achieve competitive performance over other plain PETL methods with lower training memory overhead. Our code is publicly available at: https://github.com/Paranioar/UniPT.

翻译：参数高效迁移学习（PETL）通过微调少量参数，是适配预训练模型至下游领域的有效策略。为降低内存需求，近期PETL研究聚焦于更具价值的内存高效特性。本文指出，现有最优方法的可扩展性、适应性与泛化性受限于对特定预训练骨干网络的结构依赖性与相关性。为此，我们提出一种新型内存高效PETL策略——通用并行微调（UniPT），以缓解上述缺陷。具体而言，通过轻量级可学习并行网络促进迁移过程，该网络包含：1）并行交互模块，解耦顺序连接并与预训练网络分离式处理中间激活；2）置信度聚合模块，自适应学习跨层特征融合的最优策略。我们在多种骨干网络（如T5、VSE∞、CLIP4Clip、Clip-ViL及MDETR）上评估UniPT，涵盖视觉-语言及纯自然语言处理任务。基于18个数据集的全面消融实验验证，UniPT不仅能显著降低内存消耗并超越最优竞争者，还能在更低训练内存开销下达到与常规PETL方法相匹敌的性能。我们的代码已开源：https://github.com/Paranioar/UniPT。