Sparse neural networks have shown similar or better generalization performance than their dense counterparts while having higher parameter efficiency. This has motivated a number of works to learn, induce, or search for high performing sparse networks. While reports of quality or efficiency gains are impressive, standard baselines are lacking, therefore hindering having reliable comparability and reproducibility across methods. In this work, we provide an evaluation approach and a naive Random Search baseline method for finding good sparse configurations. We apply Random Search on the node space of an overparameterized network with the goal of finding better initialized sparse sub-networks that are positioned more advantageously in the loss landscape. We record sparse network post-training performances at various levels of sparsity and compare against both their fully connected parent networks and random sparse configurations at the same sparsity levels. We observe that for this architecture search task, initialized sparse networks found by Random Search neither perform better nor converge more efficiently than their random counterparts. Thus we conclude that Random Search may be viewed as a suitable neutral baseline for sparsity search methods.

翻译：稀疏神经网络在保持更高参数效率的同时，展现出与稠密网络相似甚至更优的泛化性能。这一发现推动了众多研究致力于学习、诱导或搜索高性能稀疏网络。尽管关于质量或效率提升的报告令人印象深刻，但由于缺乏标准基线方法，不同方法之间的可靠可比性和可重复性受到制约。本研究提出一种评估方法及基于朴素随机搜索的基线方案，用于寻找优质稀疏配置。我们在过参数化网络的节点空间上应用随机搜索，旨在定位初始化更优且损失曲面中位置更有利的稀疏子网络。我们记录了不同稀疏度下稀疏网络训练后的性能，并将其分别与全连接父网络及相同稀疏度的随机稀疏配置进行对比。结果表明，在此架构搜索任务中，随机搜索发现的初始化稀疏网络既未表现出更优性能，也未实现更高效的收敛。因此，我们认为随机搜索可被视为稀疏性搜索方法中适宜的中性基线。