Large-scale pretraining followed by task-specific finetuning has achieved great success in various NLP tasks. Since finetuning all parameters of large pretrained models poses substantial computational and memory challenges, several efficient finetuning methods have been developed. Among them, low-rank adaptation (LoRA), which finetunes low-rank incremental update matrices on top of frozen pretrained weights, has proven particularly effective. Nonetheless, LoRA's uniform rank assignment across all layers, along with its reliance on an exhaustive search to find the best rank, leads to high computation costs and suboptimal finetuning performance. To address these limitations, we introduce AutoLoRA, a meta learning based framework for automatically identifying the optimal rank of each LoRA layer. AutoLoRA associates each rank-1 matrix in a low-rank update matrix with a selection variable, which determines whether the rank-1 matrix should be discarded. A meta learning based method is developed to learn these selection variables. The optimal rank is determined by thresholding the values of these variables. Our comprehensive experiments on natural language understanding, generation, and sequence labeling demonstrate the effectiveness of AutoLoRA.

翻译：大规模预训练后接任务特定微调已在各种自然语言处理任务中取得巨大成功。由于微调大型预训练模型的所有参数会带来巨大的计算和内存挑战，研究者已开发出多种高效微调方法。其中，低秩适配（LoRA）通过在冻结的预训练权重之上微调低秩增量更新矩阵，已被证明特别有效。然而，LoRA在所有层中采用统一的秩分配，且需要穷举搜索以找到最佳秩，导致计算成本高昂且微调性能次优。为解决这些限制，我们提出AutoLoRA——一种基于元学习的框架，用于自动识别每个LoRA层的最佳秩。AutoLoRA将低秩更新矩阵中的每个秩-1矩阵与一个选择变量关联，该变量决定是否应丢弃该秩-1矩阵。我们开发了一种基于元学习的方法来学习这些选择变量，通过阈值化这些变量的值来确定最佳秩。我们在自然语言理解、生成和序列标注任务上的全面实验证明了AutoLoRA的有效性。