Spiking Transformers have gained considerable attention because they achieve both the energy efficiency of Spiking Neural Networks (SNNs) and the high capacity of Transformers. However, the existing Spiking Transformer architectures, derived from Artificial Neural Networks (ANNs), exhibit a notable architectural gap, resulting in suboptimal performance compared to their ANN counterparts. Manually discovering optimal architectures is time-consuming. To address these limitations, we introduce AutoST, a training-free NAS method for Spiking Transformers, to rapidly identify high-performance Spiking Transformer architectures. Unlike existing training-free NAS methods, which struggle with the non-differentiability and high sparsity inherent in SNNs, we propose to utilize Floating-Point Operations (FLOPs) as a performance metric, which is independent of model computations and training dynamics, leading to a stronger correlation with performance. Our extensive experiments show that AutoST models outperform state-of-the-art manually or automatically designed SNN architectures on static and neuromorphic datasets. Full code, model, and data are released for reproduction.

翻译：尖峰Transformer因兼具尖峰神经网络（SNN）的能效优势与Transformer的高容量特性而备受关注。然而，当前基于人工神经网络（ANN）推导的尖峰Transformer架构存在显著的结构性差距，导致其性能相较ANN同类架构欠佳。人工探索最优架构耗时巨大。针对上述局限，我们提出AutoST——一种面向尖峰Transformer的无训练神经架构搜索方法，可快速识别高性能尖峰Transformer架构。与现有无训练NAS方法受限于SNN非可微性与高稀疏性不同，我们提出使用浮点运算数（FLOPs）作为性能评估指标。该指标独立于模型计算与训练动态，与性能呈现更强的相关性。大量实验表明，在静态数据集与神经形态数据集上，AutoST模型均优于当前最先进的人工/自动设计的SNN架构。为促进可复现性，我们已开源完整代码、模型与数据。