We show that the representation cost of fully connected neural networks with homogeneous nonlinearities - which describes the implicit bias in function space of networks with $L_2$-regularization or with losses such as the cross-entropy - converges as the depth of the network goes to infinity to a notion of rank over nonlinear functions. We then inquire under which conditions the global minima of the loss recover the `true' rank of the data: we show that for too large depths the global minimum will be approximately rank 1 (underestimating the rank); we then argue that there is a range of depths which grows with the number of datapoints where the true rank is recovered. Finally, we discuss the effect of the rank of a classifier on the topology of the resulting class boundaries and show that autoencoders with optimal nonlinear rank are naturally denoising.

翻译：我们证明，具有齐次非线性的全连接神经网络的表示成本——该成本描述了具有$L_2$正则化或交叉熵等损失的网络在函数空间中的隐式偏差——随着网络深度趋于无穷，收敛为非线性函数的一种秩概念。随后，我们探究损失函数的全局最小值在何种条件下能够恢复数据的“真实”秩：研究表明，当深度过大时，全局最小值将近似为秩1（低估了真实秩）；我们进一步论证，存在一个随数据点数量增加而增长的深度范围，在此范围内真实秩得以恢复。最后，我们讨论了分类器的秩对分类边界拓扑结构的影响，并证明具有最优非线性秩的自编码器天然具有去噪能力。