The implicit bias towards solutions with favorable properties is believed to be a key reason why neural networks trained by gradient-based optimization can generalize well. While the implicit bias of gradient flow has been widely studied for homogeneous neural networks (including ReLU and leaky ReLU networks), the implicit bias of gradient descent is currently only understood for smooth neural networks. Therefore, implicit bias in non-smooth neural networks trained by gradient descent remains an open question. In this paper, we aim to answer this question by studying the implicit bias of gradient descent for training two-layer fully connected (leaky) ReLU neural networks. We showed that when the training data are nearly-orthogonal, for leaky ReLU activation function, gradient descent will find a network with a stable rank that converges to $1$, whereas for ReLU activation function, gradient descent will find a neural network with a stable rank that is upper bounded by a constant. Additionally, we show that gradient descent will find a neural network such that all the training data points have the same normalized margin asymptotically. Experiments on both synthetic and real data backup our theoretical findings.

翻译：摘要：梯度优化训练得到的神经网络能够泛化良好的一个关键原因在于其倾向于收敛到具有优良性质的解。尽管梯度流的隐式偏差已在齐次神经网络（包括ReLU和Leaky ReLU网络）中被广泛研究，但目前梯度下降的隐式偏差仅被平滑神经网络所理解。因此，在非平滑神经网络中，梯度下降训练的隐式偏差仍是一个开放性问题。本文旨在通过研究梯度下降训练两层全连接（泄漏）ReLU神经网络的隐式偏差来回答这一问题。我们证明：当训练数据近似正交时，对于Leaky ReLU激活函数，梯度下降将找到一个稳定秩收敛到1的网络；而对于ReLU激活函数，梯度下降将找到一个稳定秩受常数上界约束的神经网络。此外，我们证明梯度下降将找到一个使所有训练数据点渐近具有相同归一化边距的神经网络。合成数据与真实数据上的实验结果均支撑了我们的理论发现。