Res-Tuning introduces a flexible and efficient paradigm for model tuning, showing that tuners decoupled from the backbone network can achieve performance comparable to traditional methods. Existing methods commonly construct the tuner as a set of trainable low-rank decomposition matrices, positing that a low-rank subspace suffices for adapting pre-trained foundational models to new scenarios. In this work, we present an advanced, efficient tuner augmented with low-rank attention, termed Res-Attn , which also adheres to the Res-Tuning framework. Res-Attn utilizes a parallel multi-head attention module equipped with low-rank projections for query, key, and value to execute streamlined attention operations. Through training this lightweight attention module, Res-Attn facilitates adaptation to new scenarios. Our extensive experiments across a range of discriminative and generative tasks showcase the superior performance of our method when compared to existing alternatives

翻译：Res-Tuning提出了一种灵活且高效的模型微调范式，证明从主干网络中解耦的调谐器能够实现与传统方法相当的性能。现有方法通常将调谐器构建为一组可训练的低秩分解矩阵，认为低秩子空间足以将预训练基础模型适应到新场景。在本文中，我们提出了一种增强型高效调谐器——Res-Attn，其配备了低秩注意力机制，并同样遵循Res-Tuning框架。Res-Attn采用并行多头注意力模块，该模块对查询、键和值使用低秩投影，以实现精简的注意力操作。通过训练这个轻量级注意力模块，Res-Attn能够适应新场景。我们在多种判别性和生成性任务上的广泛实验表明，与现有替代方法相比，我们的方法展现出优越的性能。