Neural network compression has been an increasingly important subject, due to its practical implications in terms of reducing the computational requirements and its theoretical implications, as there is an explicit connection between compressibility and the generalization error. Recent studies have shown that the choice of the hyperparameters of stochastic gradient descent (SGD) can have an effect on the compressibility of the learned parameter vector. Even though these results have shed some light on the role of the training dynamics over compressibility, they relied on unverifiable assumptions and the resulting theory does not provide a practical guideline due to its implicitness. In this study, we propose a simple modification for SGD, such that the outputs of the algorithm will be provably compressible without making any nontrivial assumptions. We consider a one-hidden-layer neural network trained with SGD and we inject additive heavy-tailed noise to the iterates at each iteration. We then show that, for any compression rate, there exists a level of overparametrization (i.e., the number of hidden units), such that the output of the algorithm will be compressible with high probability. To achieve this result, we make two main technical contributions: (i) we build on a recent study on stochastic analysis and prove a 'propagation of chaos' result with improved rates for a class of heavy-tailed stochastic differential equations, and (ii) we derive strong-error estimates for their Euler discretization. We finally illustrate our approach on experiments, where the results suggest that the proposed approach achieves compressibility with a slight compromise from the training and test error.

翻译：神经网络压缩因其在降低计算需求方面的实际意义以及与泛化误差之间的显式关联而日益成为重要课题。近期研究表明，随机梯度下降（SGD）超参数的选择会影响学习参数向量的可压缩性。尽管这些结果揭示了训练动态对可压缩性的作用，但其依赖不可验证的假设，且理论推导因隐含性而缺乏实践指导。本研究提出SGD的简单改进方案，使算法输出在无任何非平凡假设下具有可证明的可压缩性。我们考虑使用SGD训练的单隐层神经网络，并在每次迭代中向迭代点注入加性重尾噪声。随后证明：对任意压缩率，存在过参数化水平（即隐单元数量），使得算法输出能以高概率实现可压缩。为获得该结果，我们做出两项主要技术贡献：(i) 基于近期随机分析研究，证明一类重尾随机微分方程的改进速率“混沌传播”结果；(ii) 推导其欧拉离散化的强误差估计。最后通过实验验证该方法的有效性，结果表明所提方法在训练误差与测试误差略有折衷的情况下实现了可压缩性。