While many phenomena in physics and engineering are formally high-dimensional, their long-time dynamics often live on a lower-dimensional manifold. The present work introduces an autoencoder framework that combines implicit regularization with internal linear layers and $L_2$ regularization (weight decay) to automatically estimate the underlying dimensionality of a data set, produce an orthogonal manifold coordinate system, and provide the mapping functions between the ambient space and manifold space, allowing for out-of-sample projections. We validate our framework's ability to estimate the manifold dimension for a series of datasets from dynamical systems of varying complexities and compare to other state-of-the-art estimators. We analyze the training dynamics of the network to glean insight into the mechanism of low-rank learning and find that collectively each of the implicit regularizing layers compound the low-rank representation and even self-correct during training. Analysis of gradient descent dynamics for this architecture in the linear case reveals the role of the internal linear layers in leading to faster decay of a "collective weight variable" incorporating all layers, and the role of weight decay in breaking degeneracies and thus driving convergence along directions in which no decay would occur in its absence. We show that this framework can be naturally extended for applications of state-space modeling and forecasting by generating a data-driven dynamic model of a spatiotemporally chaotic partial differential equation using only the manifold coordinates. Finally, we demonstrate that our framework is robust to hyperparameter choices.

翻译：尽管物理学和工程学中的许多现象在形式上具有高维特征，但其长期动力学往往存在于低维流形上。本文提出了一种自编码器框架，该框架将隐式正则化与内部线性层及$L_2$正则化（权重衰减）相结合，以自动估计数据集的潜在维度、生成正交流形坐标系，并提供环境空间与流形空间之间的映射函数，从而实现样本外投影。我们验证了该框架在估计一系列不同复杂度动力系统数据集流形维度方面的能力，并与其它最先进的估计算法进行了比较。通过分析网络的训练动力学，我们深入探究了低秩学习机制的机理，发现每个隐式正则化层会共同增强低秩表示，甚至在训练过程中进行自我修正。在线性情况下对该架构的梯度下降动力学分析揭示了内部线性层的作用：它们能加速包含所有层的“集体权重变量”的衰减；同时，权重衰减在打破退化性方面发挥作用，从而在无权重衰减时不会发生衰减的方向上驱动收敛。我们证明，该框架可自然地扩展至状态空间建模与预测应用，通过仅使用流形坐标生成了时空混沌偏微分方程的数据驱动动态模型。最后，我们展示了该框架对超参数选择的鲁棒性。