The existence of "lottery tickets" arXiv:1803.03635 at or near initialization raises the tantalizing question of whether large models are necessary in deep learning, or whether sparse networks can be quickly identified and trained without ever training the dense models that contain them. However, efforts to find these sparse subnetworks without training the dense model ("pruning at initialization") have been broadly unsuccessful arXiv:2009.08576. We put forward a theoretical explanation for this, based on the model's effective parameter count, $p_\text{eff}$, given by the sum of the number of non-zero weights in the final network and the mutual information between the sparsity mask and the data. We show the Law of Robustness of arXiv:2105.12806 extends to sparse networks with the usual parameter count replaced by $p_\text{eff}$, meaning a sparse neural network which robustly interpolates noisy data requires a heavily data-dependent mask. We posit that pruning during and after training outputs masks with higher mutual information than those produced by pruning at initialization. Thus two networks may have the same sparsities, but differ in effective parameter count based on how they were trained. This suggests that pruning near initialization may be infeasible and explains why lottery tickets exist, but cannot be found fast (i.e. without training the full network). Experiments on neural networks confirm that information gained during training may indeed affect model capacity.

翻译：在初始化阶段或附近存在的“彩票”现象（arXiv:1803.03635）引发了一个诱人的问题：深度学习是否必须依赖大型模型，还是可以快速识别并训练稀疏网络，而无需训练包含这些子网络的密集模型？然而，在不训练密集模型的情况下寻找这些稀疏子网络（即“初始化剪枝”）的尝试普遍未能成功（arXiv:2009.08576）。我们基于模型的有效参数计数$p_\text{eff}$（即最终网络中非零权重的数量与稀疏掩码和数据之间的互信息之和）提出了一种理论解释。我们证明arXiv:2105.12806中的鲁棒性定律可以推广至稀疏网络，只需将通常的参数计数替换为$p_\text{eff}$，这意味着能够鲁棒地插值噪声数据的稀疏神经网络必须依赖于高度数据相关的掩码。我们假设，在训练过程中及训练后进行剪枝所产生的掩码，其互信息高于初始化剪枝产生的掩码。因此，两个网络可能具有相同的稀疏度，但由于训练方式不同，其有效参数计数也会不同。这表明在初始化附近进行剪枝可能不可行，并解释了彩票现象存在但无法快速找到（即无需训练完整网络）的原因。神经网络上的实验证实，训练过程中获取的信息确实可能影响模型容量。