A neural network with one hidden layer or a two-layer network (regardless of the input layer) is the simplest feedforward neural network, whose mechanism may be the basis of more general network architectures. However, even to this type of simple architecture, it is also a ``black box''; that is, it remains unclear how to interpret the mechanism of its solutions obtained by the back-propagation algorithm and how to control the training process through a deterministic way. This paper systematically studies the first problem by constructing universal function-approximation solutions. It is shown that, both theoretically and experimentally, the training solution for the one-dimensional input could be completely understood, and that for a higher-dimensional input can also be well interpreted to some extent. Those results pave the way for thoroughly revealing the black box of two-layer ReLU networks and advance the understanding of deep ReLU networks.

翻译：单隐藏层神经网络（或称双层网络，不计输入层）是最简单的前馈神经网络，其工作机制可能构成更一般网络架构的基础。然而，即使对于这种简单架构，它仍然是一个“黑箱”；也就是说，我们尚不清楚如何解释通过反向传播算法获得的解的工作机制，以及如何通过确定性方式控制训练过程。本文通过构建通用函数逼近解，系统研究了第一个问题。研究证明，从理论和实验两方面，一维输入的训练解可以被完全理解，而更高维输入的训练解在一定程度上也能得到较好解释。这些结果为彻底揭示双层ReLU网络的黑箱机制铺平了道路，并推进了对深度ReLU网络的理解。