Recent efforts to scale Transformer models have demonstrated rapid progress across a wide range of tasks (Wei et al., 2022). However, fine-tuning these models for downstream tasks is expensive due to their large parameter counts. Parameter-efficient fine-tuning (PEFT) approaches have emerged as a viable alternative by allowing us to fine-tune models by updating only a small number of parameters. In this work, we propose a general framework for parameter efficient fine-tuning (PEFT), based on structured unrestricted-rank matrices (SURM) which can serve as a drop-in replacement for popular approaches such as Adapters and LoRA. Unlike other methods like LoRA, SURMs provides more flexibility in finding the right balance between compactness and expressiveness. This is achieved by using low displacement rank matrices (LDRMs), which hasn't been used in this context before. SURMs remain competitive with baselines, often providing significant quality improvements while using a smaller parameter budget. SURMs achieve 5-7% accuracy gains on various image classification tasks while replacing low-rank matrices in LoRA. It also results in up to 12x reduction of the number of parameters in adapters (with virtually no loss in quality) on the GLUE benchmark.

翻译：近年来，扩展Transformer模型的研究在各种任务上取得了快速进展（Wei等人，2022）。然而，由于这些模型参数量巨大，针对下游任务进行微调的成本十分高昂。参数高效微调方法通过仅更新少量参数来实现模型微调，已成为一种可行的替代方案。本文提出了一种基于结构化无限制秩矩阵的参数高效微调通用框架，该框架可作为Adapter和LoRA等主流方法的即插即用替代方案。与LoRA等方法不同，SURM在紧凑性与表达能力之间提供了更灵活的平衡调节能力，这是通过采用此前未在该领域应用过的低位移秩矩阵实现的。SURM在保持与基线方法竞争力的同时，通常能以更少的参数量实现显著的性能提升。在各类图像分类任务中，SURM替换LoRA中的低秩矩阵可获得5-7%的准确率提升。在GLUE基准测试中，该方法将Adapter参数量减少高达12倍，且几乎无质量损失。