We present a novel Parameter-Efficient Fine-Tuning (PEFT) method, dubbed as Adaptive Freezing of Low Rank Adaptation (AFLoRA). Specifically, for each pre-trained frozen weight tensor, we add a parallel path of trainable low-rank matrices, namely a down-projection and an up-projection matrix, each of which is followed by a feature transformation vector. Based on a novel freezing score, we the incrementally freeze these projection matrices during fine-tuning to reduce the computation and alleviate over-fitting. Our experimental results demonstrate that we can achieve state-of-the-art performance with an average improvement of up to $0.85\%$ as evaluated on GLUE benchmark while yeilding up to $9.5\times$ fewer average trainable parameters. While compared in terms of runtime, AFLoRA can yield up to $1.86\times$ improvement as opposed to similar PEFT alternatives. Besides the practical utility of our approach, we provide insights on the trainability requirements of LoRA paths at different modules and the freezing schedule for the different projection matrices. Code will be released.

翻译：我们提出了一种新颖的参数高效微调（PEFT）方法，称为自适应冻结低秩自适应（AFLoRA）。具体而言，对于每个预训练的冻结权重张量，我们添加一条可训练的低秩矩阵并行路径，即一个下投影矩阵和一个上投影矩阵，每个矩阵之后跟随一个特征变换向量。基于一种新颖的冻结分数，我们在微调过程中逐步冻结这些投影矩阵，以减少计算量并缓解过拟合。我们的实验结果表明，在GLUE基准测试中，我们能够实现平均提升高达$0.85\%$的最新性能，同时平均可训练参数减少多达$9.5\times$。在运行时方面，与类似的PEFT替代方法相比，AFLoRA可实现高达$1.86\times$的改进。除了我们方法的实际效用外，我们还提供了关于不同模块中LoRA路径的可训练性要求以及不同投影矩阵的冻结计划的见解。代码将公开。