To better understand complexity in neural networks, we theoretically investigate the idealised phenomenon of lossless network compressibility, whereby an identical function can be implemented with fewer hidden units. In the setting of single-hidden-layer hyperbolic tangent networks, we define the rank of a parameter as the minimum number of hidden units required to implement the same function. We give efficient formal algorithms for optimal lossless compression and computing the rank of a parameter. Losslessly compressible parameters are atypical, but their existence has implications for nearby parameters. We define the proximate rank of a parameter as the rank of the most compressible parameter within a small L-infinity neighbourhood. We give an efficient greedy algorithm for bounding the proximate rank of a parameter, and show that the problem of tightly bounding the proximate rank is NP-complete. These results lay a foundation for future theoretical and empirical work on losslessly compressible parameters and their neighbours.

翻译：为了更好地理解神经网络的复杂性，我们从理论上研究了无损网络可压缩性的理想化现象，即相同的函数可以用更少的隐藏单元实现。在单隐藏层双曲正切网络的设定中，我们将参数的秩定义为实现相同函数所需的最小隐藏单元数。我们给出了用于最优无损压缩和计算参数秩的高效形式算法。无损可压缩参数是非典型的，但它们的存在对其邻近参数具有影响。我们将参数的邻近秩定义为在一个小的L-无穷邻域内最可压缩参数的秩。我们给出了一种高效的贪心算法来界定参数的邻近秩，并证明了紧密界定邻近秩的问题是NP完全的。这些结果为未来关于无损可压缩参数及其邻近参数的理论和实证工作奠定了基础。