In this paper we fully describe the trajectory of gradient flow over diagonal linear networks in the limit of vanishing initialisation. We show that the limiting flow successively jumps from a saddle of the training loss to another until reaching the minimum $\ell_1$-norm solution. This saddle-to-saddle dynamics translates to an incremental learning process as each saddle corresponds to the minimiser of the loss constrained to an active set outside of which the coordinates must be zero. We explicitly characterise the visited saddles as well as the jumping times through a recursive algorithm reminiscent of the LARS algorithm used for computing the Lasso path. Our proof leverages a convenient arc-length time-reparametrisation which enables to keep track of the heteroclinic transitions between the jumps. Our analysis requires negligible assumptions on the data, applies to both under and overparametrised settings and covers complex cases where there is no monotonicity of the number of active coordinates. We provide numerical experiments to support our findings.

翻译：本文完整描述了在初始化趋于零的极限条件下，梯度流在对角线线性网络中的演化轨迹。我们证明，极限流会依次从训练损失的鞍点跳至另一鞍点，直至达到最小ℓ1范数解。这种鞍点至鞍点动力学转化为增量学习过程——每个鞍点对应于在活动集约束下的损失极小值，而该活动集之外的坐标必须为零。我们通过递归算法明确刻画了所访问的鞍点及跳跃时间，该算法令人联想到用于计算Lasso路径的LARS算法。本文证明采用了一种便捷的弧长时间重参数化方法，从而能够追踪跳跃之间的异宿轨道转换。我们的分析对数据几乎无假设要求，适用于欠参数化和过参数化两种情形，并覆盖了活动坐标数量无单调性的复杂情况。最后通过数值实验验证了所得结论。