Theoretically exploring the advantages of neural networks might be one of the most challenging problems in the AI era. An adaptive feature program has recently been proposed to analyze feature learning, the characteristic property of neural networks, in a more abstract way. Motivated by the celebrated Le Cam equivalence, we advocate the over-parameterized sequence models to further simplify the analysis of the training dynamics of adaptive feature program and present several pieces of supporting evidence for the adaptive feature program. More precisely, after having introduced the feature error measure (FEM) to characterize the quality of the learned feature, we show that the FEM is decreasing during the training process of several concrete adaptive feature models including linear regression, single/multiple index models, etc. We believe that this hints at the potential successes of the adaptive feature program.

翻译：从理论上探索神经网络的优势可能是人工智能时代最具挑战性的问题之一。最近有人提出了一种自适应特征程序，以更抽象的方式分析神经网络的特征学习这一特性。受著名的Le Cam等价性启发，我们倡导使用过参数化的序列模型来进一步简化自适应特征程序训练动态的分析，并提出了若干支持自适应特征程序的证据。更具体地说，在引入特征误差度量（FEM）来表征所学特征的质量之后，我们证明了在包括线性回归、单/多指标模型在内的多个具体自适应特征模型的训练过程中，FEM是递减的。我们相信这暗示了自适应特征程序潜在的成功可能。