We can compare the expressiveness of neural networks that use rectified linear units (ReLUs) by the number of linear regions, which reflect the number of pieces of the piecewise linear functions modeled by such networks. However, enumerating these regions is prohibitive and the known analytical bounds are identical for networks with same dimensions. In this work, we approximate the number of linear regions through empirical bounds based on features of the trained network and probabilistic inference. Our first contribution is a method to sample the activation patterns defined by ReLUs using universal hash functions. This method is based on a Mixed-Integer Linear Programming (MILP) formulation of the network and an algorithm for probabilistic lower bounds of MILP solution sets that we call MIPBound, which is considerably faster than exact counting and reaches values in similar orders of magnitude. Our second contribution is a tighter activation-based bound for the maximum number of linear regions, which is particularly stronger in networks with narrow layers. Combined, these bounds yield a fast proxy for the number of linear regions of a deep neural network.

翻译：我们可以将使用纠正线性单元(ReLUs)的神经网络的清晰度与线性区域的数量进行比较,这些网络反映了以这种网络为模型的片断线性功能的碎片数量。 然而,列出这些地区令人望而却步,已知的分析界限对具有相同尺寸的网络来说是相同的。在这项工作中,我们根据经过训练的网络的特点和概率推断,通过经验界限来估计线性区域的数量。我们的第一个贡献是用一种方法来抽样ReLUs使用普遍散列函数定义的激活模式。这个方法基于网络的混合- Intger线性线性编程(MILP)的配制和MILP解算法,我们称之为MIPBBound,其精确计算速度大大快于精确计算和达到类似数量级的数值。我们的第二个贡献是更紧密的线性区域的最大数量,在窄层的网络中特别强。这些界限为深内线性网络的线性区域数目提供快速的代理。