In this paper, we propose a novel layer-adaptive weight-pruning approach for Deep Neural Networks (DNNs) that addresses the challenge of optimizing the output distortion minimization while adhering to a target pruning ratio constraint. Our approach takes into account the collective influence of all layers to design a layer-adaptive pruning scheme. We discover and utilize a very important additivity property of output distortion caused by pruning weights on multiple layers. This property enables us to formulate the pruning as a combinatorial optimization problem and efficiently solve it through dynamic programming. By decomposing the problem into sub-problems, we achieve linear time complexity, making our optimization algorithm fast and feasible to run on CPUs. Our extensive experiments demonstrate the superiority of our approach over existing methods on the ImageNet and CIFAR-10 datasets. On CIFAR-10, our method achieves remarkable improvements, outperforming others by up to 1.0% for ResNet-32, 0.5% for VGG-16, and 0.7% for DenseNet-121 in terms of top-1 accuracy. On ImageNet, we achieve up to 4.7% and 4.6% higher top-1 accuracy compared to other methods for VGG-16 and ResNet-50, respectively. These results highlight the effectiveness and practicality of our approach for enhancing DNN performance through layer-adaptive weight pruning. Code will be available on https://github.com/Akimoto-Cris/RD_VIT_PRUNE.

翻译：本文提出一种新颖的层自适应权重剪枝方法，用于深度神经网络（DNN），旨在解决在满足目标剪枝比例约束条件下优化输出失真最小化的挑战。我们的方法综合考虑所有层的集体影响，设计了一种层自适应剪枝方案。我们发现并利用了由多层剪枝权重引起的输出失真非常重要的可加性特性。该特性使我们能够将剪枝问题形式化为组合优化问题，并通过动态规划高效求解。通过将问题分解为子问题，我们实现了线性时间复杂度，从而使优化算法快速且可在CPU上运行。大量实验表明，我们的方法在ImageNet和CIFAR-10数据集上优于现有方法。在CIFAR-10上，我们的方法在ResNet-32、VGG-16和DenseNet-121上的top-1准确率分别比现有方法提升高达1.0%、0.5%和0.7%。在ImageNet上，对于VGG-16和ResNet-50，相比其他方法，我们分别实现了高达4.7%和4.6%的top-1准确率提升。这些结果凸显了我们的方法通过层自适应权重剪枝增强DNN性能的有效性和实用性。代码将在https://github.com/Akimoto-Cris/RD_VIT_PRUNE上发布。