Diagonal linear networks (DLNs) are a toy simplification of artificial neural networks; they consist in a quadratic reparametrization of linear regression inducing a sparse implicit regularization. In this paper, we describe the trajectory of the gradient flow of DLNs in the limit of small initialization. We show that incremental learning is effectively performed in the limit: coordinates are successively activated, while the iterate is the minimizer of the loss constrained to have support on the active coordinates only. This shows that the sparse implicit regularization of DLNs decreases with time. This work is restricted to the underparametrized regime with anti-correlated features for technical reasons.

翻译：对角线性网络(DLNs)是人工神经网络的玩具简化;它们包括线性回归的二次再修复,导致隐含的正规化程度微弱。在本文件中,我们描述了小初始化限度内DLN梯度流的轨迹。我们表明,在极限内有效实现了递增学习:坐标相继激活,而迭代是损失最小化,只能得到主动坐标的支持。这表明DLN的稀薄隐含的正规化随时间而减少。由于技术原因,这项工作仅限于具有反碳相关特征的反平衡制度。