Low-Rank Adaptation (LoRA) is the leading parameter-efficient fine-tuning method for Large Language Models (LLMs). However, the fine-tuned LLMs encounter the issue of catastrophic forgetting of the pre-trained world knowledge. To address this issue, inspired by theoretical insights of null space, we propose LoRA-Null, i.e., Low-Rank Adaptation via null space, which builds adapters initialized from the null space of the pre-trained knowledge activation. Concretely, we randomly collect a few data samples and capture their activations after passing through the LLM layer. We perform Singular Value Decomposition on the input activations to obtain their null space. We use the projection of the pre-trained weights onto the null space as the initialization for adapters. Experimental results demonstrate that this initialization approach can effectively preserve the original pre-trained world knowledge of the LLMs during fine-tuning. Additionally, if we freeze the values of the down-projection matrices during fine-tuning, it achieves even better preservation of the pre-trained world knowledge. LoRA-Null effectively preserves pre-trained world knowledge while maintaining strong fine-tuning performance, as validated by extensive experiments on LLaMA series (LLaMA2, LLaMA3, LLaMA3.1, and LLaMA3.2) across Code, Math, and Instruction Following tasks. We also provide a theoretical guarantee for the capacity of LoRA-Null to retain pre-trained knowledge. Code is in https://github.com/HungerPWAY/LoRA-Null.

翻译：低秩自适应（LoRA）是当前大语言模型（LLM）参数高效微调的主流方法。然而，微调后的LLM存在灾难性遗忘预训练世界知识的问题。为解决该问题，受零空间理论启发，我们提出LoRA-Null，即基于零空间的低秩自适应方法，该方法构建从预训练知识激活的零空间初始化的适配器。具体而言，我们随机收集少量数据样本，捕获其通过LLM层后的激活值。对输入激活值进行奇异值分解以获得其零空间。我们将预训练权重在零空间上的投影作为适配器的初始化。实验结果表明，这种初始化方法在微调过程中能有效保留LLM原有的预训练世界知识。此外，若在微调期间冻结下投影矩阵的数值，能实现更好的预训练世界知识保留效果。LoRA-Null在有效保留预训练世界知识的同时，保持了强大的微调性能，这通过在LLaMA系列模型（LLaMA2、LLaMA3、LLaMA3.1和LLaMA3.2）上针对代码、数学和指令跟随任务的广泛实验得到验证。我们还为LoRA-Null保留预训练知识的能力提供了理论保证。代码位于 https://github.com/HungerPWAY/LoRA-Null。