A class of neural networks that gained particular interest in the last years are neural ordinary differential equations (neural ODEs). We study input-output relations of neural ODEs using dynamical systems theory and prove several results about the exact embedding of maps in different neural ODE architectures in low and high dimension. The embedding capability of a neural ODE architecture can be increased by adding, for example, a linear layer, or augmenting the phase space. Yet, there is currently no systematic theory available and our work contributes towards this goal by developing various embedding results as well as identifying situations, where no embedding is possible. The mathematical techniques used include as main components iterative functional equations, Morse functions and suspension flows, as well as several further ideas from analysis. Although practically, mainly universal approximation theorems are used, our geometric dynamical systems viewpoint on universal embedding provides a fundamental understanding, why certain neural ODE architectures perform better than others.

翻译：近年来，一类备受关注的神经网络是神经常微分方程（neural ODEs）。我们利用动力系统理论研究神经ODE的输入-输出关系，并针对不同神经ODE架构在低维与高维空间中精确嵌入映射的能力，证明了若干结论。通过添加线性层或扩展相空间等方式，可增强神经ODE架构的嵌入能力。然而，目前尚缺乏系统性理论，本研究通过发展多种嵌入结果并识别无法实现嵌入的情形，为推进该目标做出了贡献。所采用的核心数学工具包括迭代函数方程、莫尔斯函数与悬链流，以及分析学中的若干其他思想。尽管实际应用中主要依赖通用逼近定理，但我们对通用嵌入的几何动力系统视角，为理解某些神经ODE架构性能更优的根本原因提供了基础性认知。