It is commonly recognized that the expressiveness of deep neural networks is contingent upon a range of factors, encompassing their depth, width, and other relevant considerations. Currently, the practical performance of the majority of deep neural networks remains uncertain. For ReLU (Rectified Linear Unit) networks with piecewise linear activations, the number of linear convex regions serves as a natural metric to gauge the network's expressivity. In this paper, we count the number of linear convex regions in deep neural networks based on ReLU. In particular, we prove that for any one-dimensional input, there exists a minimum threshold for the number of neurons required to express it. We also empirically observe that for the same network, intricate inputs hinder its capacity to express linear regions. Furthermore, we unveil the iterative refinement process of decision boundaries in ReLU networks during training. We aspire for our research to serve as an inspiration for network optimization endeavors and aids in the exploration and analysis of the behaviors exhibited by deep networks.

翻译：普遍认为，深度神经网络的表达能力取决于一系列因素，包括其深度、宽度以及其他相关考量。目前，大多数深度神经网络的实际性能仍不确定。对于采用分段线性激活函数的ReLU（修正线性单元）网络而言，线性凸区域的数量是衡量网络表达能力的一个自然指标。在本文中，我们基于ReLU统计了深度神经网络中的线性凸区域数量。具体而言，我们证明对于任何一维输入，表达该输入所需的神经元数量存在一个最小阈值。我们还通过实验观察到，对于同一网络，复杂输入会削弱其表达线性区域的能力。此外，我们揭示了训练过程中ReLU网络决策边界的迭代优化过程。我们期望我们的研究能为网络优化工作提供启发，并有助于探索和分析深度网络的行为表现。