Sparse training is one of the promising techniques to reduce the computational cost of DNNs while retaining high accuracy. In particular, N:M fine-grained structured sparsity, where only N out of consecutive M elements can be nonzero, has attracted attention due to its hardware-friendly pattern and capability of achieving a high sparse ratio. However, the potential to accelerate N:M sparse DNN training has not been fully exploited, and there is a lack of efficient hardware supporting N:M sparse training. To tackle these challenges, this paper presents a computation-efficient training scheme for N:M sparse DNNs using algorithm, architecture, and dataflow co-design. At the algorithm level, a bidirectional weight pruning method, dubbed BDWP, is proposed to leverage the N:M sparsity of weights during both forward and backward passes of DNN training, which can significantly reduce the computational cost while maintaining model accuracy. At the architecture level, a sparse accelerator for DNN training, namely SAT, is developed to neatly support both the regular dense operations and the computation-efficient N:M sparse operations. At the dataflow level, multiple optimization methods ranging from interleave mapping, pre-generation of N:M sparse weights, and offline scheduling, are proposed to boost the computational efficiency of SAT. Finally, the effectiveness of our training scheme is evaluated on a Xilinx VCU1525 FPGA card using various DNN models and datasets. Experimental results show the SAT accelerator with the BDWP sparse training method under 2:8 sparse ratio achieves an average speedup of 1.75x over that with the dense training, accompanied by a negligible accuracy loss of 0.56% on average. Furthermore, our proposed training scheme significantly improves the training throughput by 2.97~25.22x and the energy efficiency by 1.36~3.58x over prior FPGA-based accelerators.

翻译：稀疏训练是降低深度神经网络计算成本同时保持高精度的关键技术之一。其中，N:M细粒度结构化稀疏（即连续M个元素中仅有N个非零值）因其硬件友好的模式和高稀疏比率能力而备受关注。然而，N:M稀疏DNN训练的加速潜力尚未被充分挖掘，且缺乏支持N:M稀疏训练的高效硬件。为解决这些挑战，本文提出一种通过算法、架构与数据流协同设计的面向N:M稀疏DNN的高计算效率训练方案。在算法层面，提出双向权重剪枝方法BDWP，可在DNN训练的前向与反向传播中利用权重的N:M稀疏性，显著降低计算成本的同时保持模型精度。在架构层面，设计了一种面向DNN训练的稀疏加速器SAT，能够无缝支持常规稠密运算与高计算效率的N:M稀疏运算。在数据流层面，提出交织映射、N:M稀疏权重预生成与离线调度等多种优化方法，以提升SAT的计算效率。最终，在Xilinx VCU1525 FPGA开发板上使用多种DNN模型与数据集评估了本方案的有效性。实验结果表明，采用BDWP稀疏训练方法的SAT加速器在2:8稀疏比率下，相较于稠密训练实现了平均1.75倍的加速，且平均精度损失仅0.56%。此外，与基于FPGA的现有加速器相比，本方案将训练吞吐量提升2.97~25.22倍，能效提升1.36~3.58倍。