Sparse neural networks promise efficiency, yet training them effectively remains a fundamental challenge. Despite advances in pruning methods that create sparse architectures, understanding why some sparse structures are better trainable than others with the same level of sparsity remains poorly understood. Aiming to develop a systematic approach to this fundamental problem, we propose a novel theoretical framework based on the theory of graph limits, particularly graphons, that characterizes sparse neural networks in the infinite-width regime. Our key insight is that connectivity patterns of sparse neural networks induced by pruning methods converge to specific graphons as networks' width tends to infinity, which encodes implicit structural biases of different pruning methods. We postulate the Graphon Limit Hypothesis and provide empirical evidence to support it. Leveraging this graphon representation, we derive a Graphon Neural Tangent Kernel (Graphon NTK) to study the training dynamics of sparse networks in the infinite width limit. Graphon NTK provides a general framework for the theoretical analysis of sparse networks. We empirically show that the spectral analysis of Graphon NTK correlates with observed training dynamics of sparse networks, explaining the varying convergence behaviours of different pruning methods. Our framework provides theoretical insights into the impact of connectivity patterns on the trainability of various sparse network architectures.

翻译：稀疏神经网络虽有望提升效率，但其有效训练仍是一个根本性挑战。尽管剪枝方法在创建稀疏架构方面取得了进展，但为何在相同稀疏度下某些稀疏结构比其他结构更易于训练，其内在机理仍不甚明晰。为针对这一根本问题建立系统性研究方法，我们基于图极限理论（特别是图极限函数）提出了一种新颖的理论框架，用于刻画无限宽度条件下的稀疏神经网络。我们的核心洞见是：随着网络宽度趋于无穷，由剪枝方法诱导的稀疏神经网络连接模式会收敛于特定的图极限函数，这些函数编码了不同剪枝方法所隐含的结构偏好。我们提出图极限假说并提供了实证证据予以支持。利用该图极限表示，我们推导出图极限神经正切核以研究无限宽度极限下稀疏网络的训练动态。图极限神经正切核为稀疏网络的理论分析提供了一个通用框架。我们通过实证表明，图极限神经正切核的谱分析与观测到的稀疏网络训练动态具有相关性，从而解释了不同剪枝方法收敛行为的差异。本框架为连接模式对各种稀疏网络架构可训练性的影响提供了理论见解。