Spectral-domain CNNs have been shown to be more efficient than traditional spatial CNNs in terms of reducing computation complexity. However they come with a `kernel explosion' problem that, even after compression (pruning), imposes a high memory burden and off-chip bandwidth requirement for kernel access. This creates a performance gap between the potential acceleration offered by compression and actual FPGA implementation performance, especially for low-latency CNN inference. In this paper, we develop a principled approach to overcoming this performance gap and designing a low-latency, low-bandwidth, spectral sparse CNN accelerator on FPGAs. First, we analyze the bandwidth-storage tradeoff of sparse convolutional layers and locate communication bottlenecks. We then develop a dataflow for flexibly optimizing data reuse in different layers to minimize off-chip communication. Finally, we propose a novel scheduling algorithm to optimally schedule the on-chip memory access of multiple sparse kernels and minimize read conflicts. On a state-of-the-art FPGA platform, our design reduces data transfers by 42\% with DSP utilization up to 90\% and achieves inference latency of 9 ms for VGG16, compared to the baseline state-of-the-art latency of 68 ms.

翻译：谱域卷积神经网络已被证明在降低计算复杂度方面比传统空域CNN更高效。然而，这类网络存在一个"核爆炸"问题——即便经过压缩（剪枝）处理，仍会为核访问带来高内存负担和片外带宽需求。这导致压缩提供的潜在加速能力与FPGA实际实现性能之间存在差距，尤其是在低延迟CNN推理场景中。本文提出了一种克服该性能差距的原理性方法，旨在FPGA上设计低延迟、低带宽的谱域稀疏CNN加速器。首先，我们分析了稀疏卷积层的带宽-存储权衡关系，定位了通信瓶颈。进而开发了一种数据流方案，通过灵活优化不同层的数据复用来最小化片外通信。最后，我们提出一种新型调度算法，可优化调度多个稀疏核的片上内存访问，并最大限度减少读取冲突。在先进FPGA平台上，我们的设计将数据传输量降低42%，DSP利用率达90%，VGG16推理延迟为9毫秒，而基线方案的先进延迟为68毫秒。