Large language models (LLMs) have exhibited remarkable proficiency across a diverse array of natural language processing (NLP) tasks. However, adapting LLMs to downstream applications typically necessitates computationally intensive and memory-demanding fine-tuning procedures. To mitigate these burdens, parameter-efficient fine-tuning (PEFT) techniques have emerged as a promising approach to tailor LLMs with minimal computational overhead. While PEFT methods offer substantial advantages, they do not fully address the pervasive issue of bias propagation from pre-training data. In this work, we introduce Bias-Aware Low-Rank Adaptation (BA-LoRA), a novel PEFT method designed to counteract bias inheritance. BA-LoRA incorporates three distinct regularization terms: (1) consistency regularizer, (2) diversity regularizer, and (3) singular vector decomposition regularizer. These regularizers collectively aim to improve the generative models' consistency, diversity, and generalization capabilities during the fine-tuning process. Through extensive experiments on a variety of natural language understanding (NLU) and natural language generation (NLG) tasks, employing prominent LLMs such as LLaMA, Mistral, and Gemma, we demonstrate that BA-LoRA surpasses the performance of LoRA and its state-of-the-art variants. Moreover, our method effectively mitigates the deleterious effects of pre-training bias, leading to more reliable and robust model outputs. The code is available at https://github.com/cyp-jlu-ai/BA-LoRA.

翻译：大语言模型（LLMs）在各类自然语言处理（NLP）任务中展现出卓越的性能。然而，将LLMs适配至下游应用通常需要计算密集且内存需求巨大的微调过程。为减轻这些负担，参数高效微调（PEFT）技术作为一种有前景的方法应运而生，能够以最小的计算开销定制LLMs。尽管PEFT方法提供了显著优势，它们并未完全解决预训练数据中普遍存在的偏差传播问题。本文提出偏差感知低秩适应（BA-LoRA），一种旨在对抗偏差继承的新型PEFT方法。BA-LoRA引入了三种不同的正则化项：（1）一致性正则器，（2）多样性正则器，以及（3）奇异向量分解正则器。这些正则器共同致力于在微调过程中提升生成模型的一致性、多样性和泛化能力。通过在多种自然语言理解（NLU）和自然语言生成（NLG）任务上进行广泛实验，并采用LLaMA、Mistral和Gemma等主流LLMs，我们证明BA-LoRA在性能上超越了LoRA及其最先进的变体。此外，我们的方法有效缓解了预训练偏差的有害影响，从而产生更可靠、更鲁棒的模型输出。代码发布于 https://github.com/cyp-jlu-ai/BA-LoRA。