This paper presents a theoretical analysis of linear interpolation as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear interpolation can help by leveraging the theory of nonexpansive operators. We construct a new optimization scheme called relaxed approximate proximal point (RAPP), which is the first explicit method to achieve last iterate convergence rates for the full range of cohypomonotone problems. The construction extends to constrained and regularized settings. By replacing the inner optimizer in RAPP we rediscover the family of Lookahead algorithms for which we establish convergence in cohypomonotone problems even when the base optimizer is taken to be gradient descent ascent. The range of cohypomonotone problems in which Lookahead converges is further expanded by exploiting that Lookahead inherits the properties of the base optimizer. We corroborate the results with experiments on generative adversarial networks which demonstrates the benefits of the linear interpolation present in both RAPP and Lookahead.

翻译：本文对线性插值作为稳定（大规模）神经网络训练的一种规范化方法进行了理论分析。我们认为优化过程中的不稳定性通常由损失景观的非单调性引起，并展示了线性插值如何通过利用非扩张算子理论来缓解这一问题。我们构建了一种名为松弛近似近端点（RAPP）的新型优化方案，这是首个在全部协单调问题范围内实现最终迭代收敛率的显式方法。该构造可扩展至带约束和正则化场景。通过替换RAPP中的内部优化器，我们重新发现了Lookahead算法族，并证明了即便基础优化器采用梯度上升下降法，该算法在协单调问题中仍能收敛。进一步地，利用Lookahead继承基础优化器性质的特性，可扩展其收敛的协单调问题范围。我们通过生成对抗网络的实验验证了相关结论，证明了RAPP与Lookahead中线性插值的实际优势。