We prove a universal approximation property (UAP) for a class of ODENet and a class of ResNet, which are simplified mathematical models for deep learning systems with skip connections. The UAP can be stated as follows. Let $n$ and $m$ be the dimension of input and output data, and assume $m\leq n$. Then we show that ODENet of width $n+m$ with any non-polynomial continuous activation function can approximate any continuous function on a compact subset on $\mathbb{R}^n$. We also show that ResNet has the same property as the depth tends to infinity. Furthermore, we derive the gradient of a loss function explicitly with respect to a certain tuning variable. We use this to construct a learning algorithm for ODENet. To demonstrate the usefulness of this algorithm, we apply it to a regression problem, a binary classification, and a multinomial classification in MNIST.

翻译：我们证明了一类ODENet和一类ResNet（具有跳跃连接的深度学习系统的简化数学模型）的通用逼近性质（UAP）。该UAP可表述如下：设$n$和$m$分别为输入和输出数据的维度，且$m\leq n$。则我们证明，宽度为$n+m$且使用任意非多项式连续激活函数的ODENet能逼近$\mathbb{R}^n$上任意紧子集上的连续函数。同时证明，当深度趋于无穷时，ResNet具有相同性质。此外，我们显式推导了损失函数关于某个调优变量的梯度，并据此构建了ODENet的学习算法。为验证该算法的有效性，将其应用于回归问题、二分类问题及MNIST数据集上的多分类问题。