This paper introduces Standard Basis LoRA (SBoRA), a novel parameter-efficient fine-tuning approach for Large Language Models that builds upon the pioneering works of Low-Rank Adaptation (LoRA) and Orthogonal Adaptation. SBoRA further reduces the computational and memory requirements of LoRA while enhancing learning performance. By leveraging orthogonal standard basis vectors to initialize one of the low-rank matrices, either A or B, SBoRA enables regional weight updates and memory-efficient fine-tuning. This approach gives rise to two variants, SBoRA-FA and SBoRA-FB, where only one of the matrices is updated, resulting in a sparse update matrix with a majority of zero rows or columns. Consequently, the majority of the fine-tuned model's weights remain unchanged from the pre-trained weights. This characteristic of SBoRA, wherein regional weight updates occur, is reminiscent of the modular organization of the human brain, which efficiently adapts to new tasks. Our empirical results demonstrate the superiority of SBoRA-FA over LoRA in various fine-tuning tasks, including commonsense reasoning and arithmetic reasoning. Furthermore, we evaluate the effectiveness of QSBoRA on quantized LLaMA models of varying scales, highlighting its potential for efficient adaptation to new tasks. Code is available at https://github.com/CityUHK-AI/SBoRA

翻译：本文提出了一种新颖的参数高效微调方法——标准基低秩自适应（SBoRA），该方法建立在低秩自适应（LoRA）与正交自适应的开创性工作基础上。SBoRA在提升学习性能的同时，进一步降低了LoRA的计算与内存需求。通过利用正交标准基向量初始化低秩矩阵A或B之一，SBoRA实现了区域权重更新与内存高效的微调。该方法衍生出SBoRA-FA和SBoRA-FB两种变体，其中仅更新一个矩阵，从而生成具有大量零行或零列的稀疏更新矩阵。因此，微调后模型的大部分权重保持与预训练权重一致。SBoRA这种区域权重更新的特性，令人联想到人脑高效适应新任务的模块化组织机制。我们的实证结果表明，在常识推理和算术推理等多种微调任务中，SBoRA-FA均优于LoRA。此外，我们评估了QSBoRA在不同规模量化LLaMA模型上的有效性，凸显了其高效适应新任务的潜力。代码发布于https://github.com/CityUHK-AI/SBoRA