Recent studies have demonstrated that many layers are functionally redundant in large language models (LLMs), enabling model compression by removing these layers to reduce inference cost. While such approaches can improve efficiency, indiscriminate layer pruning often results in significant performance degradation. In this paper, we propose GRASP (Gradient-based Retention of Adaptive Singular Parameters), a novel compression framework that mitigates this issue by preserving sensitivity-aware singular values. Unlike direct layer pruning, GRASP leverages gradient-based attribution on a small calibration dataset to adaptively identify and retain critical singular components. By replacing redundant layers with only a minimal set of parameters, GRASP achieves efficient compression while maintaining strong performance with minimal overhead. Experiments across multiple LLMs show that GRASP consistently outperforms existing compression methods, achieving 90% of the original model's performance under 20% compression ratio.

翻译：近期研究表明，大型语言模型（LLM）中的许多层在功能上存在冗余，这为通过移除这些层来降低推理成本的模型压缩提供了可能。尽管此类方法能提升效率，但无差别的层剪枝往往导致显著的性能下降。本文提出GRASP（基于梯度的自适应奇异参数保留），一种通过保留敏感度感知奇异值来缓解该问题的新型压缩框架。与直接层剪枝不同，GRASP利用小型校准数据集上的梯度归因来自适应识别并保留关键奇异分量。通过仅用极少量参数替换冗余层，GRASP在维持强劲性能与最小开销的同时实现了高效压缩。在多个LLM上的实验表明，GRASP始终优于现有压缩方法，在20%压缩率下仍能保持原始模型90%的性能。