Parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), enable fast specialization of large pre-trained models to different downstream applications. However, this process often leads to catastrophic forgetting of the model's prior domain knowledge. We address this issue with LaLoRA, a weight-space regularization technique that applies a Laplace approximation to Low-Rank Adaptation. Our approach estimates the model's confidence in each parameter and constrains updates in high-curvature directions, preserving prior knowledge while enabling efficient target-domain learning. By applying the Laplace approximation only to the LoRA weights, the method remains lightweight. We evaluate LaLoRA by fine-tuning a Llama model for mathematical reasoning and demonstrate an improved learning-forgetting trade-off, which can be directly controlled via the method's regularization strength. We further explore different loss landscape curvature approximations for estimating parameter confidence, analyze the effect of the data used for the Laplace approximation, and study robustness across hyperparameters.

翻译：参数高效微调方法，如低秩适应（LoRA），能够快速将大型预训练模型适配至不同的下游应用。然而，该过程常导致模型先前领域知识的灾难性遗忘。我们通过LaLoRA解决此问题，这是一种权重空间正则化技术，将拉普拉斯近似应用于低秩适应。我们的方法估计模型对每个参数的置信度，并约束高曲率方向上的更新，从而在实现高效目标域学习的同时保留先验知识。通过仅对LoRA权重应用拉普拉斯近似，该方法保持了轻量化特性。我们通过微调Llama模型进行数学推理来评估LaLoRA，并展示了改进的学习-遗忘权衡，该权衡可通过方法的正则化强度直接控制。我们进一步探索了用于估计参数置信度的不同损失景观曲率近似方法，分析了用于拉普拉斯近似的数据的影响，并研究了超参数间的鲁棒性。