Pruning neural networks before training has received increasing interest due to its potential to reduce training time and memory. One popular method is to prune the connections based on a certain metric, but it is not entirely clear what metric is the best choice. Recent advances in neural tangent kernel (NTK) theory suggest that the training dynamics of large enough neural networks is closely related to the spectrum of the NTK. Motivated by this finding, we propose to prune the connections that have the least influence on the spectrum of the NTK. This method can help maintain the NTK spectrum, which may help align the training dynamics to that of its dense counterpart. However, one possible issue is that the fixed-weight-NTK corresponding to a given initial point can be very different from the NTK corresponding to later iterates during the training phase. We further propose to sample multiple realizations of random weights to estimate the NTK spectrum. Note that our approach is weight-agnostic, which is different from most existing methods that are weight-dependent. In addition, we use random inputs to compute the fixed-weight-NTK, making our method data-agnostic as well. We name our foresight pruning algorithm Neural Tangent Kernel Spectrum-Aware Pruning (NTK-SAP). Empirically, our method achieves better performance than all baselines on multiple datasets.

翻译：在训练前对神经网络进行剪枝因能减少训练时间和内存而受到日益关注。一种常见方法基于特定指标剪除连接，但何种指标为最优选择尚未完全明确。神经正切核（NTK）理论的最新进展表明，足够大的神经网络的训练动态与NTK谱密切相关。受此启发，我们提出剪除对NTK谱影响最小的连接。该方法有助于维持NTK谱，从而可能使训练动态与密集网络的训练动态对齐。然而，一个潜在问题是：对应给定初始点的固定权重NTK可能与训练阶段后期迭代对应的NTK存在显著差异。我们进一步提出对随机权重的多种实现进行采样以估计NTK谱。值得注意的是，我们的方法是权重无关的，这与大多数现有方法的权重依赖特性不同。此外，我们使用随机输入计算固定权重NTK，使得方法同样与数据无关。我们将这种前瞻性剪枝算法命名为神经正切核谱感知剪枝（NTK-SAP）。实验表明，该方法在多个数据集上均优于所有基线方法。