This study presents an ensemble technique, SPQ (SVD-Pruning-Quantization), for large language model (LLM) compression that combines variance-retained singular value decomposition (SVD), activation-based pruning, and post-training linear quantization. Each component targets a different source of inefficiency: i) pruning removes redundant neurons in MLP layers, ii) SVD reduces attention projections into compact low-rank factors, iii) and 8-bit quantization uniformly compresses all linear layers. At matched compression ratios, SPQ outperforms individual methods (SVD-only, pruning-only, or quantization-only) in perplexity, demonstrating the benefit of combining complementary techniques. Applied to LLaMA-2-7B, SPQ achieves up to 75% memory reduction while maintaining or improving perplexity (e.g., WikiText-2 5.47 to 4.91) and preserving accuracy on downstream benchmarks such as C4, TruthfulQA, and GSM8K. Compared to strong baselines like GPTQ and SparseGPT, SPQ offers competitive perplexity and accuracy while using less memory (6.86 GB vs. 7.16 GB for GPTQ). Moreover, SPQ improves inference throughput over GPTQ, achieving up to a 1.9x speedup, which further enhances its practicality for real-world deployment. The effectiveness of SPQ's robust compression through layer-aware and complementary compression techniques may provide practical deployment of LLMs in memory-constrained environments. Code is available at: https://github.com/JiaminYao/SPQ_LLM_Compression/

翻译：本研究提出了一种用于大语言模型压缩的集成技术SPQ，它结合了保留方差的奇异值分解、基于激活的剪枝以及训练后线性量化。每个组件针对不同的低效来源：i) 剪枝移除MLP层中的冗余神经元，ii) SVD将注意力投影分解为紧凑的低秩因子，iii) 8位量化则均匀压缩所有线性层。在相同的压缩比下，SPQ在困惑度上优于单一方法，证明了结合互补技术的优势。应用于LLaMA-2-7B时，SPQ实现了高达75%的内存减少，同时保持或改善了困惑度，并在下游基准测试中保持了准确性。与GPTQ和SparseGPT等强基线相比，SPQ在提供有竞争力的困惑度与准确性的同时，使用了更少的内存。此外，SPQ的推理吞吐量优于GPTQ，实现了最高1.9倍的加速，这进一步增强了其在现实世界部署中的实用性。SPQ通过层感知和互补的压缩技术实现的有效压缩，可能为在内存受限环境中实际部署大语言模型提供方案。代码发布于：https://github.com/JiaminYao/SPQ_LLM_Compression/