Given the ever-increasing size of modern neural networks, the significance of sparse architectures has surged due to their accelerated inference speeds and minimal memory demands. When it comes to global pruning techniques, Iterative Magnitude Pruning (IMP) still stands as a state-of-the-art algorithm despite its simple nature, particularly in extremely sparse regimes. In light of the recent finding that the two successive matching IMP solutions are linearly connected without a loss barrier, we propose Sparse Weight Averaging with Multiple Particles (SWAMP), a straightforward modification of IMP that achieves performance comparable to an ensemble of two IMP solutions. For every iteration, we concurrently train multiple sparse models, referred to as particles, using different batch orders yet the same matching ticket, and then weight average such models to produce a single mask. We demonstrate that our method consistently outperforms existing baselines across different sparsities through extensive experiments on various data and neural network structures.

翻译：鉴于现代神经网络规模不断增大，稀疏架构因其加速推理速度和极低内存需求而重要性激增。在全局剪枝技术中，迭代幅度剪枝（IMP）尽管方法简单，但在极度稀疏场景下仍是最先进算法。基于最新发现——两个连续匹配的IMP解之间无损失障碍线性连接，我们提出稀疏权重平均与多粒子（SWAMP）方法，该方法是对IMP的简单改进，其性能可媲美两个IMP解的集成。在每次迭代中，我们同时训练多个稀疏模型（称为粒子），这些模型使用不同的批处理顺序但相同的匹配票证，然后对这些模型进行权重平均以生成单一掩码。通过在多种数据和神经网络结构上的大量实验，我们证明该方法在不同稀疏度下均持续优于现有基线。