In this paper, an algorithm for approximate evaluation of back-propagation in DNN training is considered, which we term Approximate Outer Product Gradient Descent with Memory (Mem-AOP-GD). The Mem-AOP-GD algorithm implements an approximation of the stochastic gradient descent by considering only a subset of the outer products involved in the matrix multiplications that encompass backpropagation. In order to correct for the inherent bias in this approximation, the algorithm retains in memory an accumulation of the outer products that are not used in the approximation. We investigate the performance of the proposed algorithm in terms of DNN training loss under two design parameters: (i) the number of outer products used for the approximation, and (ii) the policy used to select such outer products. We experimentally show that significant improvements in computational complexity as well as accuracy can indeed be obtained through Mem-AOPGD.

翻译：在本文中,考虑了DNN培训中对后推法进行近似评估的算法,我们称之为“内存的近似外产品渐变源(Mem-AOP-GD)”,Mem-AOP-GD算法仅考虑包含后推法的矩阵乘数所涉外部产品的一个子集,以近似中固有的偏差,该算法在记忆中保留了在近似中未使用的外部产品的累积。我们根据两个设计参数调查了DNN培训损失的拟议算法的绩效:(一) 近似所用外部产品的数量,和(二) 选择此类外部产品的政策。我们实验性地表明,通过Mem-AOPGD确实可以实现计算复杂性和准确性方面的重大改进。