Dynamic Sparse Training (DST) is a rapidly evolving area of research that seeks to optimize the sparse initialization of a neural network by adapting its topology during training. It has been shown that under specific conditions, DST is able to outperform dense models. The key components of this framework are the pruning and growing criteria, which are repeatedly applied during the training process to adjust the network's sparse connectivity. While the growing criterion's impact on DST performance is relatively well studied, the influence of the pruning criterion remains overlooked. To address this issue, we design and perform an extensive empirical analysis of various pruning criteria to better understand their impact on the dynamics of DST solutions. Surprisingly, we find that most of the studied methods yield similar results. The differences become more significant in the low-density regime, where the best performance is predominantly given by the simplest technique: magnitude-based pruning. The code is provided at https://github.com/alooow/fantastic_weights_paper

翻译：动态稀疏训练（DST）是一个快速发展的研究领域，旨在通过训练过程中调整网络拓扑结构来优化神经网络的稀疏初始化。研究表明，在特定条件下，DST能够超越稠密模型的性能。该框架的核心组件是剪枝与增长准则，它们在训练过程中被反复应用以调整网络的稀疏连接结构。尽管增长准则对DST性能的影响已得到较为充分的研究，但剪枝准则的作用仍被忽视。为解决这一问题，我们设计并开展了针对多种剪枝准则的大规模实证分析，以深入理解其对DST解决方案动态特性的影响。令人惊讶的是，我们发现大多数研究方法产生了相似的结果。差异在低密度区域变得更为显著，此时最佳性能主要由最简技术——基于幅度的剪枝——实现。代码参见https://github.com/alooow/fantastic_weights_paper