Convolutional Neural Networks (CNNs) are hard to deploy on edge devices due to its high computation and storage complexities. As a common practice for model compression, network pruning consists of two major categories: unstructured and structured pruning, where unstructured pruning constantly performs better. However, unstructured pruning presents a structured pattern at high pruning rates, which limits its performance. To this end, we propose a Rank-based PruninG (RPG) method to maintain the ranks of sparse weights in an adversarial manner. In each step, we minimize the low-rank approximation error for the weight matrices using singular value decomposition, and maximize their distance by pushing the weight matrices away from its low rank approximation. This rank-based optimization objective guides sparse weights towards a high-rank topology. The proposed method is conducted in a gradual pruning fashion to stabilize the change of rank during training. Experimental results on various datasets and different tasks demonstrate the effectiveness of our algorithm in high sparsity. The proposed RPG outperforms the state-of-the-art performance by 1.13% top-1 accuracy on ImageNet in ResNet-50 with 98% sparsity. The codes are available at https://github.com/huawei-noah/Efficient-Computing/tree/master/Pruning/RPG and https://gitee.com/mindspore/models/tree/master/research/cv/RPG.

翻译：卷积神经网络（CNN）因其高计算和存储复杂度而难以部署在边缘设备上。作为模型压缩的常见方法，网络剪枝主要分为非结构化剪枝和结构化剪枝两大类，其中非结构化剪枝通常表现更优。然而，在高剪枝率下，非结构化剪枝会呈现结构化模式，从而限制其性能。为此，我们提出一种基于秩的剪枝（RPG）方法，以对抗性方式维持稀疏权重的秩。在每个步骤中，我们通过奇异值分解最小化权重矩阵的低秩近似误差，并通过将权重矩阵推离其低秩近似来最大化其距离。这种基于秩的优化目标引导稀疏权重向高秩拓扑结构发展。所提方法以渐进式剪枝方式执行，以稳定训练过程中秩的变化。在多种数据集和不同任务上的实验结果表明，我们的算法在高稀疏度下具有有效性。所提出的RPG在ResNet-50上以98%稀疏度实现ImageNet top-1准确率比当前最先进方法提升1.13%。代码已开源在https://github.com/huawei-noah/Efficient-Computing/tree/master/Pruning/RPG 和 https://gitee.com/mindspore/models/tree/master/research/cv/RPG。