Existing works on the expressive power of neural networks typically assume real parameters and exact operations. In this work, we study the expressive power of quantized networks under discrete fixed-point parameters and inexact fixed-point operations with round-off errors. We first provide a necessary condition and a sufficient condition on fixed-point arithmetic and activation functions for quantized networks to represent all fixed-point functions from fixed-point vectors to fixed-point numbers. Then, we show that various popular activation functions satisfy our sufficient condition, e.g., Sigmoid, ReLU, ELU, SoftPlus, SiLU, Mish, and GELU. In other words, networks using those activation functions are capable of representing all fixed-point functions. We further show that our necessary condition and sufficient condition coincide under a mild condition on activation functions: e.g., for an activation function $σ$, there exists a fixed-point number $x$ such that $σ(x)=0$. Namely, we find a necessary and sufficient condition for a large class of activation functions. We lastly show that even quantized networks using binary weights in $\{-1,1\}$ can also represent all fixed-point functions for practical activation functions.

翻译：现有关于神经网络表达能力的研究通常假设参数为实数且运算精确。本文研究在离散定点参数及含舍入误差的非精确定点运算下，量化网络的表达能力。我们首先给出量化网络能够表示所有从定点向量到定点数的定点函数的必要条件与充分条件，该条件涉及定点算术运算与激活函数。随后证明多种常用激活函数均满足该充分条件，例如Sigmoid、ReLU、ELU、SoftPlus、SiLU、Mish与GELU。换言之，采用这些激活函数的网络能够表示所有定点函数。我们进一步证明：在激活函数满足温和条件时（例如存在定点数$x$使得$σ(x)=0$），所提必要条件与充分条件完全等价，从而为一大类激活函数建立了表达能力判定的充要条件。最后证明即使采用$\{-1,1\}$二值权重的量化网络，对于实际使用的激活函数同样能够表示所有定点函数。