With the increasingly powerful performances and enormous scales of pretrained models, promoting parameter efficiency in fine-tuning has become a crucial need for effective and efficient adaptation to various downstream tasks. One representative line of fine-tuning methods is Orthogonal Fine-tuning (OFT), which rigorously preserves the angular distances within the parameter space to preserve the pretrained knowledge. Despite the empirical effectiveness, OFT still suffers low parameter efficiency at $\mathcal{O}(d^2)$ and limited capability of downstream adaptation. Inspired by Givens rotation, in this paper, we proposed quasi-Givens Orthogonal Fine-Tuning (qGOFT) to address the problems. We first use $\mathcal{O}(d)$ Givens rotations to accomplish arbitrary orthogonal transformation in $SO(d)$ with provable equivalence, reducing parameter complexity from $\mathcal{O}(d^2)$ to $\mathcal{O}(d)$. Then we introduce flexible norm and relative angular adjustments under soft orthogonality regularization to enhance the adaptation capability of downstream semantic deviations. Extensive experiments on various tasks and pretrained models validate the effectiveness of our methods.

翻译：随着预训练模型性能日益强大且规模不断增长，提升微调过程的参数效率已成为有效适应各类下游任务的关键需求。正交微调（OFT）作为代表性微调方法之一，通过严格保持参数空间内的角距离来保留预训练知识。尽管实证效果显著，OFT仍面临参数效率低下（达$\mathcal{O}(d^2)$量级）及下游适应能力有限的问题。受Givens旋转启发，本文提出准Givens正交微调（qGOFT）以解决上述问题。我们首先利用$\mathcal{O}(d)$个Givens旋转实现$SO(d)$空间中任意正交变换的可证明等价转换，将参数复杂度从$\mathcal{O}(d^2)$降至$\mathcal{O}(d)$。随后引入基于软正交正则化的灵活范数与相对角度调整机制，以增强对下游语义偏差的适应能力。在多种任务与预训练模型上的大量实验验证了本方法的有效性。