Neural network pruning is useful for discovering efficient, high-performing subnetworks within pre-trained, dense network architectures. More often than not, it involves a three-step process -- pre-training, pruning, and re-training -- that is computationally expensive, as the dense model must be fully pre-trained. While previous work has revealed through experiments the relationship between the amount of pre-training and the performance of the pruned network, a theoretical characterization of such dependency is still missing. Aiming to mathematically analyze the amount of dense network pre-training needed for a pruned network to perform well, we discover a simple theoretical bound in the number of gradient descent pre-training iterations on a two-layer, fully-connected network, beyond which pruning via greedy forward selection [61] yields a subnetwork that achieves good training error. Interestingly, this threshold is shown to be logarithmically dependent upon the size of the dataset, meaning that experiments with larger datasets require more pre-training for subnetworks obtained via pruning to perform well. Lastly, we empirically validate our theoretical results on a multi-layer perceptron trained on MNIST.

翻译：神经网络剪枝有助于在预训练的密集网络架构中发现高效、高性能的子网络。通常，这涉及三个步骤——预训练、剪枝和再训练——由于密集模型必须完全预训练，因此该过程计算成本高昂。尽管先前的工作通过实验揭示了预训练量与剪枝网络性能之间的关系，但这种依赖关系的理论刻画仍然缺失。为了数学上分析剪枝网络良好性能所需的密集网络预训练量，我们发现在两层全连接网络上，存在一个梯度下降预训练迭代次数的简单理论界限。超过该界限后，通过贪婪前向选择[61]进行剪枝将得到一个能够实现良好训练误差的子网络。有趣的是，该阈值被证明与数据集大小呈对数依赖关系，这意味着对于更大的数据集，通过剪枝获得的子网络需要更多预训练才能表现良好。最后，我们在MNIST上训练的多层感知机上实证验证了我们的理论结果。