Transformer-based models have achieved remarkable performance in NLP tasks. However, their structural characteristics-multiple layers and attention heads-introduce efficiency challenges in inference and deployment. To address these challenges, various pruning methods have recently been proposed. Notably, gradient-based methods using Head Importance Scores (HIS) have gained traction for interpretability, efficiency, and ability to identify redundant heads. However, HIS alone has limitations as it captures only the gradient-driven contribution, overlooking the diversity of attention patterns. To overcome these limitations, we introduce a novel pruning criterion, HIES (Head Importance-Entropy Score), which integrates head importance scores with attention entropy, providing complementary evidence on per-head contribution. Empirically, HIES-based pruning yields up to 15.2% improvement in model quality and 2.04x improvement in stability over HIS-only methods, enabling substantial model compression without sacrificing either accuracy or stability. Code will be released upon publication.

翻译：基于Transformer的模型在自然语言处理任务中取得了卓越性能。然而，其结构特性——多层架构与注意力头机制——在推理与部署中引入了效率挑战。为应对这些挑战，近期已提出多种剪枝方法。值得注意的是，基于梯度、使用头部重要性评分的方法因其可解释性、高效性以及识别冗余头的能力而备受关注。然而，仅依赖头部重要性评分存在局限，因其仅捕捉梯度驱动的贡献，忽视了注意力模式的多样性。为克服这些局限，本文提出一种新颖的剪枝准则HIES（头部重要性-熵评分），该准则将头部重要性评分与注意力熵相结合，为每个注意力头的贡献提供互补性证据。实验表明，基于HIES的剪枝方法相比仅使用重要性评分的方法，在模型质量上最高提升15.2%，稳定性提升2.04倍，实现了显著的模型压缩，同时不牺牲准确率或稳定性。代码将在论文发表时开源。