To better understand complexity in neural networks, we theoretically investigate the idealised phenomenon of lossless network compressibility, whereby an identical function can be implemented with a smaller network. We give an efficient formal algorithm for optimal lossless compression in the setting of single-hidden-layer hyperbolic tangent networks. To measure lossless compressibility, we define the rank of a parameter as the minimum number of hidden units required to implement the same function. Losslessly compressible parameters are atypical, but their existence has implications for nearby parameters. We define the proximate rank of a parameter as the rank of the most compressible parameter within a small $L^\infty$ neighbourhood. Unfortunately, detecting nearby losslessly compressible parameters is not so easy: we show that bounding the proximate rank is an NP-complete problem, using a reduction from Boolean satisfiability via a geometric problem involving covering points in the plane with small squares. These results underscore the computational complexity of measuring neural network complexity, laying a foundation for future theoretical and empirical work in this direction.

翻译：为了更深入地理解神经网络的复杂性，我们从理论上研究了无损网络压缩的理想化现象，即可以用更小的网络实现相同的函数。我们针对单隐藏层双曲正切网络提出了一种高效的形式化最优无损压缩算法。为了度量无损压缩性，我们将参数秩定义为实现相同函数所需的最小隐藏单元数量。无损可压缩参数虽不典型，但其存在会对邻近参数产生影响。我们将参数的近邻秩定义为在较小的$L^\infty$邻域内最具压缩性参数的秩。遗憾的是，检测邻近的无损可压缩参数并非易事：我们通过布尔可满足性问题归约到平面内用小正方形覆盖点的几何问题，证明有界近邻秩是NP完全问题。这些结果揭示了度量神经网络复杂性的计算难度，为未来该方向的理论与实证研究奠定了基础。