The increasing complexity and parameter count of Convolutional Neural Networks (CNNs) and Transformers pose challenges in terms of computational efficiency and resource demands. Pruning has been identified as an effective strategy to address these challenges by removing redundant elements such as neurons, channels, or connections, thereby enhancing computational efficiency without heavily compromising performance. This paper builds on the foundational work of Optimal Brain Damage (OBD) by advancing the methodology of parameter importance estimation using the Hessian matrix. Unlike previous approaches that rely on approximations, we introduce Optimal Brain Apoptosis (OBA), a novel pruning method that calculates the Hessian-vector product value directly for each parameter. By decomposing the Hessian matrix across network layers and identifying conditions under which inter-layer Hessian submatrices are non-zero, we propose a highly efficient technique for computing the second-order Taylor expansion of parameters. This approach allows for a more precise pruning process, particularly in the context of CNNs and Transformers, as validated in our experiments including VGG19, ResNet32, ResNet50, and ViT-B/16 on CIFAR10, CIFAR100 and Imagenet datasets. Our code is available at https://github.com/NEU-REAL/OBA.

翻译：卷积神经网络（CNN）与Transformer模型日益增长的复杂性和参数量，在计算效率与资源需求方面带来了挑战。剪枝已被证实为应对这些挑战的有效策略，通过移除神经元、通道或连接等冗余元素，可在不明显损害性能的前提下提升计算效率。本文基于最优脑损伤（OBD）的开创性工作，进一步发展了利用海森矩阵估计参数重要性的方法。不同于以往依赖近似计算的研究，我们提出了最优脑凋亡（OBA）——一种直接计算每个参数海森-向量乘积值的新型剪枝方法。通过将海森矩阵按网络层分解，并识别层间海森子矩阵非零的条件，我们提出了一种高效计算参数二阶泰勒展开的技术。该方法能够实现更精确的剪枝过程，特别是在CNN和Transformer模型中。我们在CIFAR10、CIFAR100和ImageNet数据集上对VGG19、ResNet32、ResNet50及ViT-B/16进行的实验验证了该方法的有效性。代码已开源：https://github.com/NEU-REAL/OBA。