Many empirical studies have demonstrated the performance benefits of conditional computation in neural networks, including reduced inference time and power consumption. We study the fundamental limits of neural conditional computation from the perspective of memorization capacity. For Rectified Linear Unit (ReLU) networks without conditional computation, it is known that memorizing a collection of $n$ input-output relationships can be accomplished via a neural network with $O(\sqrt{n})$ neurons. Calculating the output of this neural network can be accomplished using $O(\sqrt{n})$ elementary arithmetic operations of additions, multiplications and comparisons for each input. Using a conditional ReLU network, we show that the same task can be accomplished using only $O(\log n)$ operations per input. This represents an almost exponential improvement as compared to networks without conditional computation. We also show that the $\Theta(\log n)$ rate is the best possible. Our achievability result utilizes a general methodology to synthesize a conditional network out of an unconditional network in a computationally-efficient manner, bridging the gap between unconditional and conditional architectures.

翻译：许多实证研究证明了神经网络中条件计算在性能上的优势，包括减少推理时间和功耗。我们从记忆容量的角度研究了神经条件计算的基本极限。对于无条件计算的修正线性单元（ReLU）网络而言，已知记忆一组$n$个输入-输出关系可通过一个具有$O(\sqrt{n})$个神经元的神经网络实现。计算该神经网络的输出对每个输入需要$O(\sqrt{n})$次基本算术运算（加法、乘法和比较）。通过使用条件ReLU网络，我们证明同样的任务每个输入仅需$O(\log n)$次运算即可完成。与无条件计算的网络相比，这代表了近乎指数级的改进。我们还证明$\Theta(\log n)$的速率是最优的。我们的可达性结果利用了一种通用方法，以计算高效的方式将无条件网络综合为条件网络，从而架起了无条件架构与条件架构之间的桥梁。