Low-Rank Adaptation (LoRA) is the leading parameter-efficient fine-tuning method for Large Language Models (LLMs), but it still suffers from catastrophic forgetting. Recent work has shown that specialized LoRA initialization can alleviate catastrophic forgetting. There are currently two approaches to LoRA initialization aimed at preventing knowledge forgetting during fine-tuning: (1) making residual weights close to pre-trained weights, and (2) ensuring the space of LoRA initialization is orthogonal to pre-trained knowledge. The former is what current methods strive to achieve, while the importance of the latter is not sufficiently recognized. We find that the space of LoRA initialization is the key to preserving pre-trained knowledge rather than the residual weights. Existing methods like MiLoRA propose making the LoRA initialization space orthogonal to pre-trained weights. However, MiLoRA utilizes the null space of pre-trained weights. Compared to pre-trained weights, the input activations of pre-trained knowledge take into account the parameters of all previous layers as well as the input data, while pre-trained weights only contain information from the current layer. Moreover, we find that the effective ranks of input activations are much smaller than those of pre-trained weights. Thus, the null space of activations is more accurate and contains less pre-trained knowledge information compared to that of weights. Based on these, we introduce LoRA-Null, our proposed method that initializes LoRA in the null space of activations. Experimental results show that LoRA-Null effectively preserves the pre-trained world knowledge of LLMs while achieving good fine-tuning performance, as evidenced by extensive experiments. Code is available at {https://github.com/HungerPWAY/LoRA-Null}.

翻译：低秩适应（LoRA）是大语言模型（LLMs）主流的参数高效微调方法，但仍存在灾难性遗忘问题。近期研究表明，专门的LoRA初始化可以缓解灾难性遗忘。目前存在两种旨在防止微调过程中知识遗忘的LoRA初始化方法：（1）使残差权重接近预训练权重；（2）确保LoRA初始化空间与预训练知识正交。现有方法致力于实现前者，而后者的重要性尚未得到充分认识。我们发现，LoRA初始化空间才是保留预训练知识的关键，而非残差权重。现有方法如MiLoRA提出使LoRA初始化空间与预训练权重正交，但MiLoRA利用的是预训练权重的零空间。相较于预训练权重，预训练知识的输入激活不仅包含当前层参数，还考虑了所有前序层的参数及输入数据，而预训练权重仅包含当前层信息。此外，我们发现输入激活的有效秩远小于预训练权重的有效秩。因此，与权重的零空间相比，激活的零空间更为精确且包含更少的预训练知识信息。基于此，我们提出LoRA-Null方法，将LoRA初始化于激活的零空间中。实验结果表明，LoRA-Null在保持良好微调性能的同时，能有效保留LLMs的预训练世界知识，大量实验数据证实了这一点。代码发布于{https://github.com/HungerPWAY/LoRA-Null}。