Rectified Linear Units (ReLU) have become the main model for the neural units in current deep learning systems. This choice has been originally suggested as a way to compensate for the so called vanishing gradient problem which can undercut stochastic gradient descent (SGD) learning in networks composed of multiple layers. Here we provide analytical results on the effects of ReLUs on the capacity and on the geometrical landscape of the solution space in two-layer neural networks with either binary or real-valued weights. We study the problem of storing an extensive number of random patterns and find that, quite unexpectedly, the capacity of the network remains finite as the number of neurons in the hidden layer increases, at odds with the case of threshold units in which the capacity diverges. Possibly more important, a large deviation approach allows us to find that the geometrical landscape of the solution space has a peculiar structure: while the majority of solutions are close in distance but still isolated, there exist rare regions of solutions which are much more dense than the similar ones in the case of threshold units. These solutions are robust to perturbations of the weights and can tolerate large perturbations of the inputs. The analytical results are corroborated by numerical findings.

翻译：修正线性单元（ReLU）已成为当前深度学习系统中神经单元的主要模型。这一选择最初被提出作为补偿所谓梯度消失问题的方式，该问题可能削弱多层网络中的随机梯度下降（SGD）学习。本文针对二值或实值权重的两层神经网络，提供了ReLU对容量及解空间几何景观影响的解析结果。我们研究了存储大量随机模式的问题，并意外发现：当隐藏层神经元数量增加时，网络容量保持有限，这与阈值单元（其容量发散）的情况截然不同。或许更为重要的是，通过大偏差方法，我们发现解空间的几何景观具有特殊结构：尽管大多数解在距离上接近但彼此孤立，却存在罕见的稠密解区域，其密集程度远超阈值单元的类似区域。这些解对权重扰动具有鲁棒性，并能容忍输入的大幅扰动。解析结果得到了数值模拟的验证。