Spiking Neural Networks (SNNs) are amenable to deployment on edge devices and neuromorphic hardware due to their lower dissipation. Recently, SNN-based transformers have garnered significant interest, incorporating attention mechanisms akin to their counterparts in Artificial Neural Networks (ANNs) while demonstrating excellent performance. However, deploying large spiking transformer models on resource-constrained edge devices such as mobile phones, still poses significant challenges resulted from the high computational demands of large uncompressed high-precision models. In this work, we introduce a novel heterogeneous quantization method for compressing spiking transformers through layer-wise quantization. Our approach optimizes the quantization of each layer using one of two distinct quantization schemes, i.e., uniform or power-of-two quantification, with mixed bit resolutions. Our heterogeneous quantization demonstrates the feasibility of maintaining high performance for spiking transformers while utilizing an average effective resolution of 3.14-3.67 bits with less than a 1% accuracy drop on DVS Gesture and CIFAR10-DVS datasets. It attains a model compression rate of 8.71x-10.19x for standard floating-point spiking transformers. Moreover, the proposed approach achieves a significant energy reduction of 5.69x, 8.72x, and 10.2x while maintaining high accuracy levels of 85.3%, 97.57%, and 80.4% on N-Caltech101, DVS-Gesture, and CIFAR10-DVS datasets, respectively.

翻译：脉冲神经网络（SNNs）因其较低的功耗特性，适合部署在边缘设备和神经形态硬件上。近年来，基于SNN的Transformer模型引起了广泛关注，它们引入了与人工神经网络（ANNs）中类似的注意力机制，同时展现出优异的性能。然而，将大型脉冲Transformer模型部署在手机等资源受限的边缘设备上仍面临重大挑战，这主要源于未压缩的高精度大模型的高计算需求。本文提出一种新颖的异构量化方法，通过分层量化来压缩脉冲Transformer模型。我们的方法采用两种不同的量化方案（即均匀量化或二次幂量化）之一，并结合混合比特精度，对每一层进行量化优化。实验表明，在DVS Gesture和CIFAR10-DVS数据集上，我们的异构量化方法能以平均3.14-3.67比特的有效分辨率维持脉冲Transformer的高性能，且准确率下降小于1%。该方法对标准浮点型脉冲Transformer实现了8.71倍至10.19倍的模型压缩率。此外，所提方法在N-Caltech101、DVS-Gesture和CIFAR10-DVS数据集上分别实现了5.69倍、8.72倍和10.2倍的显著能耗降低，同时保持了85.3%、97.57%和80.4%的高准确率。