Pruning refers to the elimination of trivial weights from neural networks. The sub-networks within an overparameterized model produced after pruning are often called Lottery tickets. This research aims to generate winning lottery tickets from a set of lottery tickets that can achieve similar accuracy to the original unpruned network. We introduce a novel winning ticket called Cyclic Overlapping Lottery Ticket (COLT) by data splitting and cyclic retraining of the pruned network from scratch. We apply a cyclic pruning algorithm that keeps only the overlapping weights of different pruned models trained on different data segments. Our results demonstrate that COLT can achieve similar accuracies (obtained by the unpruned model) while maintaining high sparsities. We show that the accuracy of COLT is on par with the winning tickets of Lottery Ticket Hypothesis (LTH) and, at times, is better. Moreover, COLTs can be generated using fewer iterations than tickets generated by the popular Iterative Magnitude Pruning (IMP) method. In addition, we also notice COLTs generated on large datasets can be transferred to small ones without compromising performance, demonstrating its generalizing capability. We conduct all our experiments on Cifar-10, Cifar-100 & TinyImageNet datasets and report superior performance than the state-of-the-art methods.

翻译：剪枝是指从神经网络中去除冗余权重。剪枝后过参数化模型中产生的子网络通常被称为彩票假设。本研究旨在从一组彩票假设中生成能够达到与原始未剪枝网络相近准确率的获胜彩票。我们通过数据分割与剪枝网络的循环重训练，提出了一种称为循环重叠彩票假设（COLT）的新型获胜彩票。我们采用循环剪枝算法，仅保留在不同数据片段上训练所得剪枝模型的重叠权重。实验结果表明，COLT在保持高稀疏度的同时能够达到与未剪枝模型相当的准确率。我们证明COLT的准确率与彩票假设理论（LTH）的获胜彩票相当，有时甚至更优。此外，相较于流行的迭代幅度剪枝（IMP）方法，生成COLT所需的迭代次数更少。值得注意的是，在大数据集上生成的COLT可以迁移至小数据集而不损失性能，这体现了其泛化能力。我们在Cifar-10、Cifar-100和TinyImageNet数据集上进行了全面实验，报告的性能均优于现有最先进方法。