Mixed-precision neural networks (MPNNs) that enable the use of just enough data width for a deep learning task promise significant advantages of both inference accuracy and computing overhead. FPGAs with fine-grained reconfiguration capability can adapt the processing with distinct data width and models, and hence, can theoretically unleash the potential of MPNNs. Nevertheless, commodity DPUs on FPGAs mostly emphasize generality and have limited support for MPNNs especially the ones with lower data width. In addition, primitive DSPs in FPGAs usually have much larger data width than that is required by MPNNs and haven't been sufficiently co-explored with MPNNs yet. To this end, we propose an open source MPNN accelerator design framework specifically tailored for FPGAs. In this framework, we have a systematic DSP-packing algorithm to pack multiple lower data width MACs in a single primitive DSP and enable efficient implementation of MPNNs. Meanwhile, we take DSP packing efficiency into consideration with MPNN quantization within a unified neural network architecture search (NAS) framework such that it can be aware of the DSP overhead during quantization and optimize the MPNN performance and accuracy concurrently. Finally, we have the optimized MPNN fine-tuned to a fully pipelined neural network accelerator template based on HLS and make best use of available resources for higher performance. Our experiments reveal the resulting accelerators produced by the proposed framework can achieve overwhelming advantages in terms of performance, resource utilization, and inference accuracy for MPNNs when compared with both handcrafted counterparts and prior hardware-aware neural network accelerators on FPGAs.

翻译：混合精度神经网络（MPNN）能够为深度学习任务使用恰好足够的位宽，在推理精度与计算开销两方面均具有显著优势。具有细粒度可重构能力的FPGA可适应不同位宽和模型的处理方式，因此理论上能够释放MPNN的潜力。然而，FPGA上的商用DPU大多强调通用性，对MPNN（尤其是低位宽MPNN）的支持有限。此外，FPGA中的原始DSP位宽通常远超MPNN所需，且尚未与MPNN进行充分协同探索。为此，我们提出一个专门针对FPGA的开源MPNN加速器设计框架。该框架采用系统性DSP打包算法，将多个低位宽乘累加运算打包到单个原始DSP中，从而实现MPNN的高效部署。同时，我们在统一的神经网络架构搜索（NAS）框架中，将DSP打包效率与MPNN量化协同考虑，使量化过程能够感知DSP开销，并同步优化MPNN的性能与精度。最后，我们基于HLS将优化后的MPNN微调至全流水线神经网络加速器模板，充分利用可用资源以提升性能。实验表明，与手工设计的FPGA加速器以及先前的硬件感知神经网络加速器相比，本框架生成的加速器在MPNN的性能、资源利用率和推理精度方面均具有压倒性优势。