We establish the fundamental limits in the approximation of Lipschitz functions by deep ReLU neural networks with finite-precision weights. Specifically, three regimes, namely under-, over-, and proper quantization, in terms of minimax approximation error behavior as a function of network weight precision, are identified. This is accomplished by deriving nonasymptotic tight lower and upper bounds on the minimax approximation error. Notably, in the proper-quantization regime, neural networks exhibit memory-optimality in the approximation of Lipschitz functions. Deep networks have an inherent advantage over shallow networks in achieving memory-optimality. We also develop the notion of depth-precision tradeoff, showing that networks with high-precision weights can be converted into functionally equivalent deeper networks with low-precision weights, while preserving memory-optimality. This idea is reminiscent of sigma-delta analog-to-digital conversion, where oversampling rate is traded for resolution in the quantization of signal samples. We improve upon the best-known ReLU network approximation results for Lipschitz functions and describe a refinement of the bit extraction technique which could be of independent general interest.

翻译：我们建立了具有有限精度权重的深度ReLU神经网络逼近Lipschitz函数的基本极限。具体而言，根据极小化极大逼近误差随网络权重精度变化的行为，识别出三种机制：欠量化、过量化和适当量化。这一结论通过推导极小化极大逼近误差的非渐近紧致下界与上界而得出。值得注意的是，在适当量化机制下，神经网络在逼近Lipschitz函数时表现出内存最优性。深度网络在实现内存最优性方面具有优于浅层网络的固有优势。我们还提出了深度-精度权衡的概念，表明具有高精度权重的网络可转换为功能等价的低精度权重深层网络，同时保持内存最优性。该思想类似于Sigma-Delta模数转换中的过采样率与信号样本量化分辨率之间的权衡。我们改进了现有最佳的Lipschitz函数ReLU网络逼近结果，并描述了比特提取技术的改进方法，该技术可能具有独立的广泛适用性。