We propose a method named AudioFormer,which learns audio feature representations through the acquisition of discrete acoustic codes and subsequently fine-tunes them for audio classification tasks. Initially,we introduce a novel perspective by considering the audio classification task as a form of natural language understanding (NLU). Leveraging an existing neural audio codec model,we generate discrete acoustic codes and utilize them to train a masked language model (MLM),thereby obtaining audio feature representations. Furthermore,we pioneer the integration of a Multi-Positive sample Contrastive (MPC) learning approach. This method enables the learning of joint representations among multiple discrete acoustic codes within the same audio input. In our experiments,we treat discrete acoustic codes as textual data and train a masked language model using a cloze-like methodology,ultimately deriving high-quality audio representations. Notably,the MPC learning technique effectively captures collaborative representations among distinct positive samples. Our research outcomes demonstrate that AudioFormer attains significantly improved performance compared to prevailing monomodal audio classification models across multiple datasets,and even outperforms audio-visual multimodal classification models on select datasets. Specifically,our approach achieves remarkable results on datasets including AudioSet (2M,20K),and FSD50K,with performance scores of 53.9,45.1,and 65.6,respectively. We have openly shared both the code and models: https://github.com/LZH-0225/AudioFormer.git.

翻译：摘要：我们提出一种名为AudioFormer的方法，该方法通过获取离散声学编码学习音频特征表示，并针对音频分类任务进行微调。首先，我们引入新视角，将音频分类任务视为自然语言理解（NLU）任务。借助现有神经音频编解码模型生成离散声学编码，并利用这些编码训练掩码语言模型（MLM），从而获得音频特征表示。此外，我们首次整合多正样本对比学习（MPC）方法，该方法能学习同一音频输入中多个离散声学编码之间的联合表示。实验中，我们将离散声学编码视为文本数据，采用类完形填空方法训练掩码语言模型，最终获得高质量音频表示。值得注意的是，MPC学习技术有效捕捉了不同正样本间的协同表示。研究结果表明，AudioFormer在多个数据集上的性能显著优于现有单模态音频分类模型，甚至在某些数据集上超越音频-视觉多模态分类模型。具体而言，我们的方法在AudioSet（2M、20K）和FSD50K数据集上取得优异结果，性能得分分别为53.9、45.1和65.6。我们已公开分享代码与模型：https://github.com/LZH-0225/AudioFormer.git。