Analyzing the behavior of ReLU neural networks often hinges on understanding the relationships between their parameters and the functions they implement. This paper proves a new bound on function distances in terms of the so-called path-metrics of the parameters. Since this bound is intrinsically invariant with respect to the rescaling symmetries of the networks, it sharpens previously known bounds. It is also, to the best of our knowledge, the first bound of its kind that is broadly applicable to modern networks such as ResNets, VGGs, U-nets, and many more. In contexts such as network pruning and quantization, the proposed path-metrics can be efficiently computed using only two forward passes. Besides its intrinsic theoretical interest, the bound yields not only novel theoretical generalization bounds, but also a promising proof of concept for rescaling-invariant pruning.

翻译：分析ReLU神经网络的行为通常依赖于理解其参数与所实现函数之间的关系。本文基于参数所谓的路径度量，证明了函数距离的新上界。由于该上界本质上对网络的重新缩放对称性保持不变，它改进了先前已知的上界。据我们所知，这也是首个广泛适用于ResNet、VGG、U-net等现代网络的此类上界。在网络剪枝和量化等场景中，所提出的路径度量仅需两次前向传播即可高效计算。除了其内在的理论价值外，该上界不仅推导出新的理论泛化界，还为尺度不变的剪枝方法提供了有前景的概念验证。