While neural networks can be approximated by linear models as their width increases, certain properties of wide neural networks cannot be captured by linear models. In this work we show that recently proposed Neural Quadratic Models can exhibit the "catapult phase" [Lewkowycz et al. 2020] that arises when training such models with large learning rates. We then empirically show that the behaviour of neural quadratic models parallels that of neural networks in generalization, especially in the catapult phase regime. Our analysis further demonstrates that quadratic models can be an effective tool for analysis of neural networks.

翻译：尽管随着宽度增加，神经网络可被线性模型近似，但宽神经网络的某些特性无法通过线性模型捕捉。本研究表明，近期提出的神经二次模型能够呈现出[Lewkowycz等人，2020]在使用大学习率训练此类模型时出现的"弹射阶段"。我们通过实验证明，神经二次模型在泛化行为上与神经网络高度相似，特别是在弹射阶段区域。我们的分析进一步表明，二次模型可作为分析神经网络的有效工具。