The advancements in Large Language Models (LLMs) have been hindered by their substantial sizes, which necessitate LLM compression methods for practical deployment. Singular Value Decomposition (SVD) offers a promising solution for LLM compression. However, state-of-the-art SVD-based LLM compression methods have two key limitations: truncating smaller singular values may lead to higher compression loss, and the lack of update on the remaining model parameters after SVD truncation. In this work, we propose SVD-LLM, a new SVD-based LLM compression method that addresses the limitations of existing methods. SVD-LLM incorporates a truncation-aware data whitening strategy to ensure a direct mapping between singular values and compression loss. Moreover, SVD-LLM adopts a layer-wise closed-form model parameter update strategy to compensate for accuracy degradation caused by SVD truncation. We evaluate SVD-LLM on a total of 11 datasets and seven models from three different LLM families at four different scales. Our results demonstrate the superiority of SVD-LLM over state-of-the-arts, especially at high model compression ratios. The source code is available at https://github.com/AIoT-MLSys-Lab/SVD-LLM.

翻译：大语言模型的快速发展受到其庞大参数规模的制约，因此需要开发高效的模型压缩方法以实现实际部署。奇异值分解为大语言模型压缩提供了可行方案。然而，现有基于奇异值分解的最优压缩方法存在两个关键局限：截断较小奇异值可能导致更高的压缩损失，以及截断后缺乏对剩余模型参数的更新机制。本文提出SVD-LLM——一种新型的基于奇异值分解的大语言模型压缩方法，旨在解决现有方案的局限性。SVD-LLM引入截断感知的数据白化策略，确保奇异值与压缩损失之间具有直接映射关系。同时，采用逐层闭式模型参数更新机制，以补偿因奇异值截断造成的精度损失。我们在11个数据集、来自三个不同大语言模型家族的七种模型及四种不同参数量级上进行了评估。实验结果表明，SVD-LLM相较于现有最优方法具有显著优势，尤其在高模型压缩比场景下表现突出。源代码已开源至 https://github.com/AIoT-MLSys-Lab/SVD-LLM。