Current deep neural networks (DNNs) are overparameterized and use most of their neuronal connections during inference for each task. The human brain, however, developed specialized regions for different tasks and performs inference with a small fraction of its neuronal connections. We propose an iterative pruning strategy introducing a simple importance-score metric that deactivates unimportant connections, tackling overparameterization in DNNs and modulating the firing patterns. The aim is to find the smallest number of connections that is still capable of solving a given task with comparable accuracy, i.e. a simpler subnetwork. We achieve comparable performance for LeNet architectures on MNIST, and significantly higher parameter compression than state-of-the-art algorithms for VGG and ResNet architectures on CIFAR-10/100 and Tiny-ImageNet. Our approach also performs well for the two different optimizers considered -- Adam and SGD. The algorithm is not designed to minimize FLOPs when considering current hardware and software implementations, although it performs reasonably when compared to the state of the art.

翻译：当前深度神经网络(DNNs)存在过参数化问题，在每项任务的推理过程中会使用大部分神经连接。相比之下，人脑针对不同任务演化出专门化的脑区，仅利用一小部分神经连接即可完成推理。我们提出一种迭代剪枝策略，引入简单的神经元重要性评分指标来停用不重要连接，从而解决DNNs的过参数化问题并调节神经发放模式。该方法旨在寻找仍能以相近精度解决特定任务的最小连接数，即更简化的子网络。在MNIST数据集上，我们对LeNet架构实现了与现有方法相当的性能；在CIFAR-10/100和Tiny-ImageNet数据集上，对VGG和ResNet架构实现了显著高于当前最优算法的参数压缩率。该方法在Adam与SGD两种不同优化器下均表现良好。尽管该算法未针对当前软硬件实现最小化浮点运算量(FLOPs)，但其性能仍与现有最优方法相当。