The evaluation of layer importance in deep learning has been an active area of research, with significant implications for model optimization and interpretability. Recently, large language models (LLMs) have gained prominence across various domains, yet limited studies have explored the functional importance and performance contributions of individual layers within LLMs, especially from the perspective of activation distribution. In this work, we propose the Activation Variance-Sparsity Score (AVSS), a novel metric combining normalized activation variance and sparsity to assess each layer's contribution to model performance. By identifying and removing approximately the lowest 25% of layers based on AVSS, we achieve over 90% of original model performance across tasks such as question answering, language modeling, and sentiment classification, indicating that these layers may be non-essential. Our approach provides a systematic method for identifying less critical layers, contributing to efficient large language model architectures.

翻译：深度学习中的层重要性评估一直是活跃的研究领域，对模型优化和可解释性具有重要意义。近年来，大语言模型（LLMs）在各个领域崭露头角，但针对LLMs中单个层的功能重要性及性能贡献的研究仍较为有限，特别是从激活分布角度进行的探索。本研究提出激活方差-稀疏性评分（AVSS），这是一种结合归一化激活方差与稀疏性的新型度量指标，用于评估各层对模型性能的贡献。通过基于AVSS识别并移除约25%的最低评分层，我们在问答、语言建模和情感分类等任务中实现了超过原模型90%的性能表现，表明这些层可能并非必需。该方法为识别非关键层提供了系统化方案，有助于构建高效的大语言模型架构。