We present a simple yet novel parameterized form of linear mapping to achieves remarkable network compression performance: a pseudo SVD called Ternary SVD (TSVD). Unlike vanilla SVD, TSVD limits the $U$ and $V$ matrices in SVD to ternary matrices form in $\{\pm 1, 0\}$. This means that instead of using the expensive multiplication instructions, TSVD only requires addition instructions when computing $U(\cdot)$ and $V(\cdot)$. We provide direct and training transition algorithms for TSVD like Post Training Quantization and Quantization Aware Training respectively. Additionally, we analyze the convergence of the direct transition algorithms in theory. In experiments, we demonstrate that TSVD can achieve state-of-the-art network compression performance in various types of networks and tasks, including current baseline models such as ConvNext, Swim, BERT, and large language model like OPT.

翻译：我们提出一种简单而新颖的线性映射参数化形式，可实现卓越的网络压缩性能：一种称为三元SVD（TSVD）的伪奇异值分解。与标准SVD不同，TSVD将SVD中的U矩阵和V矩阵限制为{±1, 0}形式的三元矩阵。这意味着在计算U(·)和V(·)时，TSVD仅需加法指令，而无需昂贵的乘法指令。我们分别提供了类似后训练量化和量化感知训练的TSVD直接转换算法与训练过渡算法。此外，我们在理论上分析了直接转换算法的收敛性。实验表明，TSVD能够在多种网络结构和任务中实现最先进的网络压缩性能，涵盖当前基线模型如ConvNeXt、Swin、BERT以及大型语言模型如OPT。