Sparse neural networks have shown similar or better generalization performance than their dense counterparts while having higher parameter efficiency. This has motivated a number of works to learn or search for high performing sparse networks. While reports of task performance or efficiency gains are impressive, standard baselines are lacking leading to poor comparability and unreliable reproducibility across methods. In this work, we propose Random Search as a baseline algorithm for finding good sparse configurations and study its performance. We apply Random Search on the node space of an overparameterized network with the goal of finding better initialized sparse sub-networks that are positioned more advantageously in the loss landscape. We record the post-training performances of the found sparse networks and at various levels of sparsity, and compare against both their fully connected parent networks and random sparse configurations at the same sparsity levels. First, we demonstrate performance at different levels of sparsity and highlight that a significant level of performance can still be preserved even when the network is highly sparse. Second, we observe that for this sparse architecture search task, initialized sparse networks found by Random Search neither perform better nor converge more efficiently than their random counterparts. Thus we conclude that Random Search may be viewed as a reasonable neutral baseline for sparsity search methods.

翻译：稀疏神经网络在保持较高参数效率的同时，展现出与密集网络相当甚至更优的泛化性能。这促使大量研究致力于学习或搜索高性能稀疏网络。尽管现有方法在任务性能或效率提升方面的报告令人瞩目，但缺乏标准化基线导致方法间的可比性差、可复现性不可靠。本文提出将随机搜索作为寻找良好稀疏配置的基线算法，并系统评估其性能。我们针对过参数化网络的节点空间执行随机搜索，旨在发现初始化位置更优（更有利于损失景观）的稀疏子网络。记录所获稀疏网络在不同稀疏度水平下的训练后性能，并与对应全连接父网络及同稀疏度随机稀疏配置进行对比。首先，我们展示了不同稀疏度水平下的性能表现，强调即使在高度稀疏网络中仍可保持显著性能水平。其次，我们观察到对于稀疏架构搜索任务，随机搜索找到的初始化稀疏网络在性能表现与收敛效率上均未优于随机稀疏配置。因此得出结论：随机搜索可作为稀疏性搜索方法合理的中性基线。