The rapid advancement in Large Language Models (LLMs) has markedly enhanced the capabilities of language understanding and generation. However, the substantial model size poses hardware challenges, affecting both memory size for serving and inference latency for token generation. To address those challenges, we propose Dependency-aware Semi-structured Sparsity (DaSS), a novel method for the recent prevalent SwiGLU-based LLMs pruning. Our approach incorporates structural dependency into the weight magnitude-based unstructured pruning. We introduce an MLP-specific pruning metric that evaluates the importance of each weight by jointly considering its magnitude and its corresponding MLP intermediate activation norms. DaSS facilitates a balance between the adaptability offered by unstructured pruning and the structural consistency inherent in dependency-based structured pruning. Empirical evaluations on Mistral and LLaMA2 model families demonstrate that DaSS not only outperforms both SparseGPT and Wanda in achieving hardware-friendly N:M sparsity patterns but also maintains the computational efficiency of Wanda.

翻译：大语言模型（LLM）的快速发展显著提升了语言理解与生成能力。然而，庞大的模型规模带来了硬件挑战，既影响服务所需的内存容量，也影响令牌生成的推理延迟。为应对这些挑战，我们提出依赖感知的半结构化稀疏性（DaSS），这是一种针对当前流行的基于SwiGLU的LLM剪枝的新方法。我们的方法将结构依赖性融入基于权重幅度的非结构化剪枝中。我们引入了一种特定于MLP的剪枝度量，通过联合考虑每个权重的幅度及其对应的MLP中间激活范数来评估其重要性。DaSS在非结构化剪枝提供的适应性与基于依赖的结构化剪枝固有的结构一致性之间取得了平衡。在Mistral和LLaMA2模型系列上的实证评估表明，DaSS不仅在实现硬件友好的N:M稀疏模式方面优于SparseGPT和Wanda，同时还保持了Wanda的计算效率。