Theoretically exploring the advantages of neural networks might be one of the most challenging problems in the AI era. An adaptive feature program has recently been proposed to analyze feature learning, the characteristic property of neural networks, in a more abstract way. Motivated by the celebrated Le Cam equivalence, we advocate the over-parameterized sequence models to further simplify the analysis of the training dynamics of adaptive feature program and present several pieces of supporting evidence for the adaptive feature program. More precisely, after having introduced the feature error measure (FEM) to characterize the quality of the learned feature, we show that the FEM is decreasing during the training process of several concrete adaptive feature models including linear regression, single/multiple index models, etc. We believe that this hints at the potential successes of the adaptive feature program.

翻译：理论上探索神经网络的优势可能是人工智能时代最具挑战性的问题之一。近期提出的自适应特征项目旨在以更抽象的方式分析神经网络的特征学习特性。受著名的Le Cam等价性启发，我们主张采用过参数化序列模型进一步简化自适应特征项目训练动力学的分析，并为该框架提供多项支持性证据。具体而言，在引入特征误差度量（FEM）以表征学习到的特征质量后，我们证明在线性回归、单/多指数模型等具体自适应特征模型的训练过程中，FEM呈现单调递减趋势。我们相信这暗示了自适应特征项目的潜在成功可能性。