This work introduces a novel approach to pruning deep learning models by using distilled data. Unlike conventional strategies which primarily focus on architectural or algorithmic optimization, our method reconsiders the role of data in these scenarios. Distilled datasets capture essential patterns from larger datasets, and we demonstrate how to leverage this capability to enable a computationally efficient pruning process. Our approach can find sparse, trainable subnetworks (a.k.a. Lottery Tickets) up to 5x faster than Iterative Magnitude Pruning at comparable sparsity on CIFAR-10. The experimental results highlight the potential of using distilled data for resource-efficient neural network pruning, model compression, and neural architecture search.

翻译：本工作提出了一种利用蒸馏数据进行深度学习模型剪枝的新方法。与主要关注架构或算法优化的传统策略不同，我们的方法重新审视了数据在这些场景中的作用。蒸馏数据集能够从更大数据集中捕获关键模式，我们展示了如何利用这一能力实现计算高效的剪枝过程。我们的方法能够在CIFAR-10数据集上，以与迭代幅度剪枝相当的稀疏度下，找到稀疏可训练子网络（即“中奖彩票”），速度最高提升5倍。实验结果凸显了使用蒸馏数据进行资源高效神经网络剪枝、模型压缩及神经架构搜索的潜力。