This paper explores the implicit bias of overparameterized neural networks of depth greater than two layers. Our framework considers a family of networks of varying depths that all have the same capacity but different implicitly defined representation costs. The representation cost of a function induced by a neural network architecture is the minimum sum of squared weights needed for the network to represent the function; it reflects the function space bias associated with the architecture. Our results show that adding linear layers to a ReLU network yields a representation cost that favors functions that can be approximated by a low-rank linear operator composed with a function with low representation cost using a two-layer network. Specifically, using a neural network to fit training data with minimum representation cost yields an interpolating function that is nearly constant in directions orthogonal to a low-dimensional subspace. This means that the learned network will approximately be a single- or multiple-index model. Our experiments show that when this active subspace structure exists in the data, adding linear layers can improve generalization and result in a network that is well-aligned with the true active subspace.

翻译：本文探讨深度大于两层的过参数化神经网络的隐式偏差。我们的框架考虑一系列不同深度的网络，它们具有相同的容量，但不同的隐式定义表示代价。由神经网络架构诱导的函数表示代价定义为网络表示该函数所需的最小平方权重之和；它反映了与架构关联的函数空间偏差。我们的结果表明，在ReLU网络中添加线性层会得到一种表示代价，该代价倾向于那些可通过低秩线性算子与两层网络中具有低表示代价的函数复合而近似的函数。具体而言，使用最小化表示代价的神经网络拟合训练数据，会生成一个插值函数，该函数在低维子空间的正交方向上几乎恒定。这意味着学习到的网络将近似为单指标或多指标模型。实验表明，当数据中存在这种活跃子空间结构时，添加线性层可以改善泛化性能，并使网络与真实活跃子空间良好对齐。