We introduce the Singular Value Representation (SVR), a new method to represent the internal state of neural networks using SVD factorization of the weights. This construction yields a new weighted graph connecting what we call spectral neurons, that correspond to specific activation patterns of classical neurons. We derive a precise statistical framework to discriminate meaningful connections between spectral neurons for fully connected and convolutional layers. To demonstrate the usefulness of our approach for machine learning research, we highlight two discoveries we made using the SVR. First, we highlight the emergence of a dominant connection in VGG networks that spans multiple deep layers. Second, we witness, without relying on any input data, that batch normalization can induce significant connections between near-kernels of deep layers, leading to a remarkable spontaneous sparsification phenomenon.

翻译：我们提出奇异值表示（Singular Value Representation, SVR），这是一种利用权重奇异值分解（SVD）来表征神经网络内部状态的新方法。该构造生成一个加权图，连接我们称之为谱神经元（spectral neurons）的单元，这些谱神经元对应经典神经元的特定激活模式。我们推导出一个精确的统计框架，用于判别全连接层和卷积层中谱神经元之间的有意义连接。为展示该方法对机器学习研究的实用价值，我们重点介绍两项基于SVR的发现：首先，我们揭示了VGG网络中跨越多个深层的主连接涌现现象；其次，无需依赖任何输入数据，我们观测到批归一化（batch normalization）能在深层近核之间诱导显著连接，从而引发引人注目的自发稀疏化现象。