Audio Spectrogram Transformer models rule the field of Audio Tagging, outrunning previously dominating Convolutional Neural Networks (CNNs). Their superiority is based on the ability to scale up and exploit large-scale datasets such as AudioSet. However, Transformers are demanding in terms of model size and computational requirements compared to CNNs. We propose a training procedure for efficient CNNs based on offline Knowledge Distillation (KD) from high-performing yet complex transformers. The proposed training schema and the efficient CNN design based on MobileNetV3 results in models outperforming previous solutions in terms of parameter and computational efficiency and prediction performance. We provide models of different complexity levels, scaling from low-complexity models up to a new state-of-the-art performance of .483 mAP on AudioSet. Source Code available at: https://github.com/fschmid56/EfficientAT

翻译：音频频谱图Transformer模型在音频标记领域占据主导地位，超越了此前占优的卷积神经网络。其优势源于能够扩展规模并利用大规模数据集（如AudioSet）。然而，相较于CNN，Transformer在模型规模和计算需求方面要求更高。我们提出了一种针对高效CNN的训练流程，该流程基于从高性能但复杂的Transformer进行离线知识蒸馏。所提出的训练方案以及基于MobileNetV3的高效CNN设计，使得模型在参数效率、计算效率和预测性能方面均优于先前方案。我们提供了不同复杂度级别的模型，从低复杂度模型到在AudioSet上达到.483 mAP的最新最优性能。源代码可在以下网址获取：https://github.com/fschmid56/EfficientAT