While large language models (LLMs) demonstrate impressive performance across various tasks, their deployment in real-world scenarios is still constrained by high computational demands. Layer-wise pruning, a commonly employed strategy to mitigate inference costs, can partially address this challenge. However, existing approaches generally depend on static heuristic rules and fail to account for the interdependencies among layers, thereby limiting the effectiveness of the pruning process. To this end, this paper proposes a game-theoretic framework that formulates layer pruning as a cooperative game in which each layer acts as a player and model performance serves as the utility. As computing exact Shapley values is computationally infeasible for large language models (LLMs), we propose using a lightweight surrogate network to estimate layer-wise marginal contributions. This network can predict LLM performance for arbitrary layer combinations at a low computational cost. Additionally, we employ stratified Monte Carlo mask sampling to further reduce the cost of Sharpley value estimation. This approach captures inter-layer dependencies and dynamically identifies critical layers for pruning. Extensive experiments demonstrate the consistent superiority of our method in terms of perplexity and zero-shot accuracy, achieving more efficient and effective layer-wise pruning for large language models.

翻译：尽管大语言模型（LLMs）在各种任务中展现出卓越性能，但其在实际场景中的部署仍受限于高昂的计算需求。分层剪枝作为降低推理成本的常用策略，能够部分缓解这一挑战。然而，现有方法通常依赖于静态启发式规则，未能考虑层间的相互依赖性，从而限制了剪枝过程的有效性。为此，本文提出一种博弈论框架，将分层剪枝建模为合作博弈：其中每一层作为参与者，模型性能则作为效用函数。由于为大语言模型（LLMs）计算精确的沙普利值在计算上不可行，我们提出使用轻量级代理网络来估计分层边际贡献。该网络能够以较低计算成本预测任意层组合下的LLM性能。此外，我们采用分层蒙特卡洛掩码采样技术，进一步降低沙普利值估计的计算开销。该方法能够捕捉层间依赖关系，并动态识别剪枝的关键层级。大量实验表明，我们的方法在困惑度和零样本准确率方面始终具有优越性，为大语言模型实现了更高效、更有效的分层剪枝。