Fine-tuning pre-trained large language models in a parameter-efficient manner is widely studied for its effectiveness and efficiency. LoRA is one of the most widely used methods, which assumes that the optimization process is essentially low dimensional. Although LoRA has demonstrated commendable performance, there remains a significant performance gap between LoRA and full fine-tuning when learning new tasks. In this work, we propose Low-Rank Adaptation with Task-Relevant Feature Enhancement(LoRATRF) for enhancing task-relevant features from the perspective of editing neural network representations. To prioritize task-relevant features, a task-aware filter that selectively extracts valuable knowledge from hidden representations for the target or current task is designed. As the experiments on a vareity of datasets including NLU, commonsense reasoning and mathematical reasoning tasks demonstrates, our method reduces 33.71% parameters and achieves better performance on a variety of datasets in comparison with SOTA low-rank methods.

翻译：以参数高效的方式微调预训练大语言模型因其有效性和高效性而被广泛研究。低秩自适应（LoRA）是最常用的方法之一，其假设优化过程本质上是低维的。尽管LoRA已展现出值得称赞的性能，但在学习新任务时，LoRA与全参数微调之间仍存在显著性能差距。本文提出基于任务相关特征增强的低秩自适应方法（LoRATRF），通过编辑神经网络表征的视角来增强任务相关特征。为优先处理任务相关特征，我们设计了一种任务感知过滤器，能够从隐藏表征中针对目标或当前任务选择性地提取有价值的知识。在自然语言理解、常识推理和数学推理等多个数据集上的实验表明，相较于最先进的低秩方法，我们的方法减少了33.71%的参数，并在多种数据集上取得了更优的性能。