We prove the first guarantees of sparse recovery for ReLU neural networks, where the sparse network weights constitute the signal to be recovered. Specifically, we study structural properties of the sparse network weights for two-layer, scalar-output networks under which a simple iterative hard thresholding algorithm recovers these weights exactly, using memory that grows linearly in the number of nonzero weights. We validate this theoretical result with simple experiments on recovery of sparse planted MLPs, MNIST classification, and implicit neural representations. Experimentally, we find performance that is competitive with, and often exceeds, a high-performing but memory-inefficient baseline based on iterative magnitude pruning. Code is available at https://github.com/voilalab/MLP-IHT.

翻译：我们首次证明了针对ReLU神经网络的稀疏恢复保证，其中稀疏网络权重构成待恢复的信号。具体而言，我们研究了两层标量输出网络中稀疏网络权重的结构特性，在此条件下，一种简单的迭代硬阈值算法能够精确恢复这些权重，且所需内存随非零权重数量线性增长。我们通过在稀疏植入多层感知机恢复、MNIST分类以及隐式神经表示上的简单实验验证了这一理论结果。实验表明，该方法的性能可与基于迭代幅度剪枝的高效但内存消耗大的基线方法相媲美，且往往更优。代码发布于https://github.com/voilalab/MLP-IHT。