We provide a new efficient version of the backpropagation algorithm, specialized to the case where the weights of the neural network being trained are sparse. Our algorithm is general, as it applies to arbitrary (unstructured) sparsity and common layer types (e.g., convolutional or linear). We provide a fast vectorized implementation on commodity CPUs, and show that it can yield speedups in end-to-end runtime experiments, both in transfer learning using already-sparsified networks, and in training sparse networks from scratch. Thus, our results provide the first support for sparse training on commodity hardware.

翻译：我们提出了一种新的高效反向传播算法，专门针对所训练神经网络权重为稀疏的情况。该算法具有通用性，适用于任意（非结构化）稀疏性及常见层类型（如卷积层或线性层）。我们在通用CPU上实现了快速向量化版本，并证明其能在端到端运行时实验中带来加速，既适用于使用已稀疏化网络的迁移学习，也适用于从头训练稀疏网络。因此，我们的结果为在通用硬件上进行稀疏训练提供了首个支持。