This paper presents a theoretical analysis of linear interpolation as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear interpolation can help by leveraging the theory of nonexpansive operators. We construct a new optimization scheme called relaxed approximate proximal point (RAPP), which is the first 1-SCLI method to achieve last iterate convergence rates for $\rho$-comonotone problems while only requiring $\rho > -\tfrac{1}{2L}$. The construction extends to constrained and regularized settings. By replacing the inner optimizer in RAPP we rediscover the family of Lookahead algorithms for which we establish convergence in cohypomonotone problems even when the base optimizer is taken to be gradient descent ascent. The range of cohypomonotone problems in which Lookahead converges is further expanded by exploiting that Lookahead inherits the properties of the base optimizer. We corroborate the results with experiments on generative adversarial networks which demonstrates the benefits of the linear interpolation present in both RAPP and Lookahead.

翻译：本文提出了一种理论分析，将线性插值作为稳定（大规模）神经网络训练的原则性方法。我们认为，优化过程中的不稳定性通常由损失景观的非单调性引起，并展示了线性插值如何通过利用非扩张算子理论来提供帮助。我们构建了一种新的优化方案，称为松弛近似近端点法（RAPP），这是首个在仅需 $\rho > -\tfrac{1}{2L}$ 条件下，为 $\rho$-共单调问题实现最后一次迭代收敛速率的 1-SCLI 方法。该构造可扩展至约束和正则化设定。通过替换 RAPP 中的内部优化器，我们重新发现了 Lookahead 算法族，并证明了即使在基础优化器采用梯度下降上升法时，该算法族在共单调问题中也能实现收敛。利用 Lookahead 继承基础优化器性质的特点，进一步扩展了其收敛的共单调问题范围。我们通过生成对抗网络实验验证了结果，表明 RAPP 和 Lookahead 中线性插值的优势。