We propose an entirely new meta-learning framework for network pruning. It is a general framework that can be theoretically applied to almost all types of networks with all kinds of pruning and has great generality and transferability. Experiments have shown that it can achieve outstanding results on many popular and representative pruning tasks (including both CNNs and Transformers). Unlike all prior works that either rely on fixed, hand-crafted criteria to prune in a coarse manner, or employ learning to prune ways that require special training during each pruning and lack generality. Our framework can learn complex pruning rules automatically via a neural network (metanetwork) and has great generality that can prune without any special training. More specifically, we introduce the newly developed idea of metanetwork from meta-learning into pruning. A metanetwork is a network that takes another network as input and produces a modified network as output. In this paper, we first establish a bijective mapping between neural networks and graphs, and then employ a graph neural network as our metanetwork. We train a metanetwork that learns the pruning strategy automatically and can transform a network that is hard to prune into another network that is much easier to prune. Once the metanetwork is trained, our pruning needs nothing more than a feedforward through the metanetwork and some standard finetuning to prune at state-of-the-art. Our code is available at https://github.com/Yewei-Liu/MetaPruning

翻译：我们提出了一种全新的网络剪枝元学习框架。该框架具有普适性，理论上可应用于几乎所有类型的网络及各类剪枝方法，并具备出色的泛化与迁移能力。实验表明，该框架在多种主流且具有代表性的剪枝任务（包括CNN与Transformer）上均能取得优异效果。与以往所有依赖固定、手工设计准则进行粗粒度剪枝，或采用需在每次剪枝时进行特殊训练且缺乏泛化性的学习式剪枝方法不同，本框架能够通过神经网络（元网络）自动学习复杂的剪枝规则，并具备强大的泛化能力，无需任何特殊训练即可完成剪枝。具体而言，我们将元学习中新近发展的元网络思想引入剪枝领域。元网络是一种以另一网络作为输入，并输出经修改网络的网络。本文首先建立了神经网络与图之间的双射映射，进而采用图神经网络作为元网络。我们训练一个能自动学习剪枝策略的元网络，它可将难以剪枝的网络转换为更易剪枝的网络。一旦元网络训练完成，我们的剪枝过程仅需通过元网络进行一次前向传播，并结合一些标准的微调，即可达到最先进的剪枝效果。代码发布于 https://github.com/Yewei-Liu/MetaPruning