While many phenomena in physics and engineering are formally high-dimensional, their long-time dynamics often live on a lower-dimensional manifold. The present work introduces an autoencoder framework that combines implicit regularization with internal linear layers and $L_2$ regularization (weight decay) to automatically estimate the underlying dimensionality of a data set, produce an orthogonal manifold coordinate system, and provide the mapping functions between the ambient space and manifold space, allowing for out-of-sample projections. We validate our framework's ability to estimate the manifold dimension for a series of datasets from dynamical systems of varying complexities and compare to other state-of-the-art estimators. We analyze the training dynamics of the network to glean insight into the mechanism of low-rank learning and find that collectively each of the implicit regularizing layers compound the low-rank representation and even self-correct during training. Analysis of gradient descent dynamics for this architecture in the linear case reveals the role of the internal linear layers in leading to faster decay of a "collective weight variable" incorporating all layers, and the role of weight decay in breaking degeneracies and thus driving convergence along directions in which no decay would occur in its absence. We show that this framework can be naturally extended for applications of state-space modeling and forecasting by generating a data-driven dynamic model of a spatiotemporally chaotic partial differential equation using only the manifold coordinates. Finally, we demonstrate that our framework is robust to hyperparameter choices.

翻译：尽管物理和工程中的许多现象在形式上具有高维特征，但其长时间动力学往往存在于低维流形上。本文提出一种自编码器框架，通过将隐式正则化与内部线性层及$L_2$正则化（权重衰减）相结合，自动估计数据集的潜在维数、生成正交流形坐标系，并提供环境空间与流形空间之间的映射函数，从而支持样本外投影。我们验证了该框架对一系列来自不同复杂度动力系统数据集进行流形维数估计的能力，并与现有最优估计器进行了比较。通过分析网络训练动力学以探究低秩学习机制，发现每个隐式正则化层会共同增强低秩表示，甚至在训练过程中实现自我修正。针对线性情形下该架构的梯度下降动力学分析揭示了内部线性层的作用——通过加速包含所有层的“集体权重变量”衰减，以及权重衰减打破退化性、驱动在无衰减时无法收敛的方向上实现收敛。我们证明该框架可自然扩展用于状态空间建模与预测，通过仅利用流形坐标为时空混沌偏微分方程生成数据驱动的动力学模型。最后，我们展示了该框架对超参数选择具有鲁棒性。