Large language models (LLM) have achieved remarkable performance across a wide range of tasks. However, their substantial parameter sizes pose significant challenges for deployment on edge devices with limited computational and memory resources. Low-rank compression is a promising approach to address this issue, as it reduces both computational and memory costs, making LLM more suitable for resource-constrained environments. Nonetheless, naïve low-rank compression methods require a significant reduction in the retained rank to achieve meaningful memory and computation savings. For a low-rank model, the ranks need to be reduced by more than half to yield efficiency gains. Such aggressive truncation, however, typically results in substantial performance degradation. To address this trade-off, we propose SkipCat, a novel low-rank compression framework that enables the use of higher ranks while achieving the same compression rates. First, we introduce an intra-layer shared low-rank projection method, where multiple matrices that share the same input use a common projection. This reduces redundancy and improves compression efficiency. Second, we propose a block skipping technique that omits computations and memory transfers for selected sub-blocks within the low-rank decomposition. These two techniques jointly enable our compressed model to retain more effective ranks under the same compression budget. Experimental results show that, without any additional fine-tuning, our method outperforms previous low-rank compression approaches by 7% accuracy improvement on zero-shot tasks under the same compression rate. These results highlight the effectiveness of our rank-maximized compression strategy in preserving model performance under tight resource constraints.

翻译：大语言模型（LLM）在广泛的任务中取得了卓越的性能。然而，其庞大的参数量对在计算和内存资源有限的边缘设备上部署构成了显著挑战。低秩压缩是一种有前景的解决方案，它通过降低计算和内存开销，使LLM更适用于资源受限的环境。然而，传统的低秩压缩方法需要大幅降低保留的秩才能实现有意义的内存和计算节省。对于低秩模型，秩需减少一半以上才能获得效率提升。但如此激进的截断通常会导致性能显著下降。为解决这一权衡问题，我们提出了SkipCat，一种新颖的低秩压缩框架，能够在相同压缩率下使用更高的秩。首先，我们引入了一种层内共享低秩投影方法，其中共享相同输入的多个矩阵使用一个公共投影，这减少了冗余并提高了压缩效率。其次，我们提出了一种块跳过技术，在低秩分解中省略选定子块的计算和内存传输。这两种技术共同使我们的压缩模型在相同压缩预算下保留更多有效秩。实验结果表明，在无需任何额外微调的情况下，我们的方法在相同压缩率下，在零样本任务上的准确率比先前的低秩压缩方法提高了7%。这些结果凸显了我们的最大化秩压缩策略在严格资源约束下保持模型性能的有效性。