To parameter-efficiently fine-tune (PEFT) large language models (LLMs), the low-rank adaptation (LoRA) method approximates the model changes $\Delta W \in \mathbb{R}^{m \times n}$ through the product of two matrices $A \in \mathbb{R}^{m \times r}$ and $B \in \mathbb{R}^{r \times n}$, where $r \ll \min(m, n)$, $A$ is initialized with Gaussian noise, and $B$ with zeros. LoRA freezes the original model $W$ and updates the "Noise & Zero" adapter, which may lead to slow convergence. To overcome this limitation, we introduce Principal Singular values and Singular vectors Adaptation (PiSSA). PiSSA shares the same architecture as LoRA, but initializes the adaptor matrices $A$ and $B$ with the principal components of the original matrix $W$, and put the remaining components into a residual matrix $W^{res} \in \mathbb{R}^{m \times n}$ which is frozen during fine-tuning. Compared to LoRA, PiSSA updates the principal components while freezing the "residual" parts, allowing faster convergence and enhanced performance. Comparative experiments of PiSSA and LoRA across 12 different models, ranging from 184M to 70B, encompassing 5 NLG and 8 NLU tasks, reveal that PiSSA consistently outperforms LoRA under identical experimental setups. On the GSM8K benchmark, Mistral-7B fine-tuned with PiSSA achieves an accuracy of 72.86%, surpassing LoRA's 67.7% by 5.16%. Due to the same architecture, PiSSA is also compatible with quantization to further reduce the memory requirement of fine-tuning. Compared to QLoRA, QPiSSA exhibits smaller quantization errors in the initial stages. Fine-tuning LLaMA-3-70B on GSM8K, QPiSSA attains an accuracy of 86.05%, exceeding the performances of QLoRA at 81.73%. Leveraging a fast SVD technique, PiSSA can be initialized in only a few seconds, presenting a negligible cost for transitioning from LoRA to PiSSA. Code is available at https://github.com/GraphPKU/PiSSA.

翻译：为高效参数微调（PEFT）大语言模型（LLMs），低秩适配（LoRA）方法通过两个矩阵 $A \in \mathbb{R}^{m \times r}$ 和 $B \in \mathbb{R}^{r \times n}$ 的乘积来近似模型变化 $\Delta W \in \mathbb{R}^{m \times n}$，其中 $r \ll \min(m, n)$，$A$ 以高斯噪声初始化，$B$ 以零初始化。LoRA 冻结原始模型 $W$ 并更新“噪声与零”适配器，这可能导致收敛缓慢。为克服此限制，我们提出了主奇异值与奇异向量适配（PiSSA）。PiSSA 与 LoRA 共享相同的架构，但使用原始矩阵 $W$ 的主成分初始化适配器矩阵 $A$ 和 $B$，并将剩余成分放入残差矩阵 $W^{res} \in \mathbb{R}^{m \times n}$ 中，该残差矩阵在微调期间被冻结。与 LoRA 相比，PiSSA 更新主成分同时冻结“残差”部分，从而实现更快的收敛和增强的性能。在涵盖 5 项自然语言生成（NLG）和 8 项自然语言理解（NLU）任务的 12 个不同模型（规模从 184M 到 70B）上进行的 PiSSA 与 LoRA 对比实验表明，在相同的实验设置下，PiSSA 始终优于 LoRA。在 GSM8K 基准测试中，使用 PiSSA 微调的 Mistral-7B 达到了 72.86% 的准确率，比 LoRA 的 67.7% 高出 5.16%。由于架构相同，PiSSA 也兼容量化以进一步降低微调的内存需求。与 QLoRA 相比，QPiSSA 在初始阶段表现出更小的量化误差。在 GSM8K 上微调 LLaMA-3-70B，QPiSSA 获得了 86.05% 的准确率，超过了 QLoRA 的 81.73%。利用快速奇异值分解（SVD）技术，PiSSA 可在几秒钟内完成初始化，为从 LoRA 切换到 PiSSA 带来的成本可忽略不计。代码发布于 https://github.com/GraphPKU/PiSSA。