Deep neural networks are superior to shallow networks in learning complex representations. As such, there is a fast-growing interest in utilizing them in large-scale settings. The training process of neural networks is already known to be time-consuming, and having a deep architecture only aggravates the issue. This process consists mostly of matrix operations, among which matrix multiplication is the bottleneck. Several sampling-based techniques have been proposed for speeding up the training time of deep neural networks by approximating the matrix products. These techniques fall under two categories: (i) sampling a subset of nodes in every hidden layer as active at every iteration and (ii) sampling a subset of nodes from the previous layer to approximate the current layer's activations using the edges from the sampled nodes. In both cases, the matrix products are computed using only the selected samples. In this paper, we evaluate the scalability of these approaches on CPU machines with limited computational resources. Making a connection between the two research directions as special cases of approximating matrix multiplications in the context of neural networks, we provide a negative theoretical analysis that shows feedforward approximation is an obstacle against scalability. We conduct comprehensive experimental evaluations that demonstrate the most pressing challenges and limitations associated with the studied approaches. We observe that the hashing-based node selection method is not scalable to a large number of layers, confirming our theoretical analysis. Finally, we identify directions for future research.

翻译：深度神经网络在学习复杂表示方面优于浅层网络。因此，在规模化场景中利用深度网络的需求快速增长。众所周知，神经网络的训练过程非常耗时，而深层架构更加剧了这一问题。该过程主要由矩阵运算组成，其中矩阵乘法是性能瓶颈。已有多种基于采样的技术通过近似矩阵乘法来加速深度神经网络的训练。这些技术分为两类：（i）在每个迭代中，从每个隐藏层采样一部分节点作为活跃节点；（ii）从上一层的节点中采样，利用采样节点的边近似计算当前层的激活值。在这两种情况下，矩阵乘积仅基于所选样本计算。本文评估了这些方法在计算资源有限的CPU机器上的可扩展性。通过将这两个研究方向建立联系，将其视为神经网络中矩阵乘法逼近的特例，我们给出了否定的理论分析，表明前馈近似是可扩展性的障碍。我们进行了全面的实验评估，揭示了与所研究方法相关的最突出的挑战和局限性。我们观察到，基于哈希的节点选择方法无法扩展到大量层，这证实了我们的理论分析。最后，我们指出了未来研究的方向。