Random masks define surprisingly effective sparse neural network models, as has been shown empirically. The resulting sparse networks can often compete with dense architectures and state-of-the-art lottery ticket pruning algorithms, even though they do not rely on computationally expensive prune-train iterations and can be drawn initially without significant computational overhead. We offer a theoretical explanation of how random masks can approximate arbitrary target networks if they are wider by a logarithmic factor in the inverse sparsity $1 / \log(1/\text{sparsity})$. This overparameterization factor is necessary at least for 3-layer random networks, which elucidates the observed degrading performance of random networks at higher sparsity. At moderate to high sparsity levels, however, our results imply that sparser networks are contained within random source networks so that any dense-to-sparse training scheme can be turned into a computationally more efficient sparse-to-sparse one by constraining the search to a fixed random mask. We demonstrate the feasibility of this approach in experiments for different pruning methods and propose particularly effective choices of initial layer-wise sparsity ratios of the random source network. As a special case, we show theoretically and experimentally that random source networks also contain strong lottery tickets.

翻译：随机掩码能够生成出奇有效的稀疏神经网络模型，这一点已通过实证得到验证。由此产生的稀疏网络通常可以与密集架构和最先进的彩票剪枝算法相竞争，尽管它们不依赖于计算成本高昂的剪枝-训练迭代，并且可以在初始时无需显著计算开销地随机生成。我们从理论上解释了随机掩码如何能够逼近任意目标网络，只要这些网络在逆稀疏度 $1 / \log(1/\text{稀疏度})$ 的对数因子范围内更宽。这一过参数化因子至少对于三层随机网络是必要的，这阐明了在更高稀疏度下随机网络性能退化的观察现象。然而，在中等至高等稀疏度水平上，我们的结果表明，更稀疏的网络包含在随机源网络中，因此任何从密集到稀疏的训练方案都可以通过将搜索限制在固定的随机掩码上，转化为计算效率更高的从稀疏到稀疏的方案。我们在不同剪枝方法的实验中展示了这种方法的可行性，并提出了随机源网络初始逐层稀疏度比率的特别有效选择。作为特例，我们从理论和实验上证明，随机源网络也包含强大的彩票。