In this article we present new results on neural networks with linear threshold activation functions. We precisely characterize the class of functions that are representable by such neural networks and show that 2 hidden layers are necessary and sufficient to represent any function representable in the class. This is a surprising result in the light of recent exact representability investigations for neural networks using other popular activation functions like rectified linear units (ReLU). We also give precise bounds on the sizes of the neural networks required to represent any function in the class. Finally, we design an algorithm to solve the empirical risk minimization (ERM) problem to global optimality for these neural networks with a fixed architecture. The algorithm's running time is polynomial in the size of the data sample, if the input dimension and the size of the network architecture are considered fixed constants. The algorithm is unique in the sense that it works for any architecture with any number of layers, whereas previous polynomial time globally optimal algorithms work only for very restricted classes of architectures. Using these insights, we propose a new class of neural networks that we call shortcut linear threshold networks. To the best of our knowledge, this way of designing neural networks has not been explored before in the literature. We show that these neural networks have several desirable theoretical properties.

翻译：本文提出了关于线性阈值激活函数神经网络的新成果。我们精确刻画了此类神经网络可表示的函数类别，并证明了在该类中表示任意函数时，两个隐藏层既是必要的也是充分的。这一结果与近期使用其他流行激活函数（如修正线性单元 ReLU）的神经网络精确可表示性研究相比，显得令人惊讶。我们还给出了在该函数类中表示任意函数所需神经网络规模的精确界限。最后，我们设计了一种算法，用于求解固定架构下此类神经网络的全局最优经验风险最小化（ERM）问题。当输入维度和网络架构规模视为固定常数时，该算法的运行时间在数据样本规模上呈多项式复杂度。该算法的独特之处在于其适用于任意层数的架构，而此前多项式时间的全局最优算法仅适用于极受限制的架构类别。基于这些见解，我们提出了一类新的神经网络——快捷线性阈值网络。据我们所知，这种神经网络设计方法此前在文献中尚未被探索。我们证明了该类神经网络具有若干理想的理论性质。