We consider the problem of one-dimensional function approximation using shallow neural networks (NN) with a rectified linear unit (ReLU) activation function and compare their training with traditional methods such as univariate Free Knot Splines (FKS). ReLU NNs and FKS span the same function space, and thus have the same theoretical expressivity. In the case of ReLU NNs, we show that their ill-conditioning degrades rapidly as the width of the network increases. This often leads to significantly poorer approximation in contrast to the FKS representation, which remains well-conditioned as the number of knots increases. We leverage the theory of optimal piecewise linear interpolants to improve the training procedure for a ReLU NN. Using the equidistribution principle, we propose a two-level procedure for training the FKS by first solving the nonlinear problem of finding the optimal knot locations of the interpolating FKS. Determining the optimal knots then acts as a good starting point for training the weights of the FKS. The training of the FKS gives insights into how we can train a ReLU NN effectively to give an equally accurate approximation. More precisely, we combine the training of the ReLU NN with an equidistribution based loss to find the breakpoints of the ReLU functions, combined with preconditioning the ReLU NN approximation (to take an FKS form) to find the scalings of the ReLU functions, leads to a well-conditioned and reliable method of finding an accurate ReLU NN approximation to a target function. We test this method on a series or regular, singular, and rapidly varying target functions and obtain good results realising the expressivity of the network in this case.

翻译：本文研究使用具有修正线性单元（ReLU）激活函数的浅层神经网络进行一维函数逼近的问题，并将其训练过程与传统方法（如单变量自由节点样条）进行比较。ReLU神经网络与自由节点样条张成相同的函数空间，因而具有相同的理论表达能力。对于ReLU神经网络，我们证明其病态性随网络宽度增加而迅速恶化，这通常导致逼近效果显著劣于自由节点样条表示——后者在节点数量增加时仍能保持良态。我们借助最优分段线性插值理论来改进ReLU神经网络的训练流程。基于等分布原理，我们提出一种双层训练策略：首先求解自由节点样条插值的最优节点位置这一非线性问题，确定的最优节点将为后续自由节点样条的权重训练提供优质初始点。自由节点样条的训练过程启示我们如何有效训练ReLU神经网络以获得同等精度的逼近。具体而言，我们将ReLU神经网络的训练与基于等分布的损失函数相结合以确定ReLU函数的断点，同时通过将ReLU神经网络逼近预处理为自由节点样条形式来确定ReLU函数的缩放系数，从而形成一种良态且可靠的训练方法，能够获得对目标函数的高精度ReLU神经网络逼近。我们在系列规则、奇异及快速变化的目标函数上测试该方法，均获得良好结果，充分实现了网络在此类场景下的表达能力。