We propose a method named AudioFormer,which learns audio feature representations through the acquisition of discrete acoustic codes and subsequently fine-tunes them for audio classification tasks. Initially,we introduce a novel perspective by considering the audio classification task as a form of natural language understanding (NLU). Leveraging an existing neural audio codec model,we generate discrete acoustic codes and utilize them to train a masked language model (MLM),thereby obtaining audio feature representations. Furthermore,we pioneer the integration of a Multi-Positive sample Contrastive (MPC) learning approach. This method enables the learning of joint representations among multiple discrete acoustic codes within the same audio input. In our experiments,we treat discrete acoustic codes as textual data and train a masked language model using a cloze-like methodology,ultimately deriving high-quality audio representations. Notably,the MPC learning technique effectively captures collaborative representations among distinct positive samples. Our research outcomes demonstrate that AudioFormer attains significantly improved performance compared to prevailing monomodal audio classification models across multiple datasets,and even outperforms audio-visual multimodal classification models on select datasets. Specifically,our approach achieves remarkable results on datasets including AudioSet (2M,20K),and FSD50K,with performance scores of 53.9,45.1,and 65.6,respectively. We have openly shared both the code and models: https://github.com/LZH-0225/AudioFormer.git.

翻译：我们提出一种名为AudioFormer的方法，该方法通过获取离散声学编码学习音频特征表示，并随后将其微调用于音频分类任务。首先，我们引入一种新颖视角，将音频分类任务视为一种自然语言理解（NLU）形式。利用现有的神经音频编解码器模型，我们生成离散声学编码，并利用这些编码训练掩码语言模型（MLM），从而获得音频特征表示。此外，我们首创性地整合了多正样本对比学习（MPC）方法。该方法能够在同一音频输入中学习多个离散声学编码之间的联合表示。在实验中，我们将离散声学编码视为文本数据，采用类似完形填空的方法训练掩码语言模型，最终推导出高质量音频表示。值得注意的是，MPC学习技术能有效捕获不同正样本之间的协作表示。我们的研究结果表明，AudioFormer在多个数据集上相比当前单模态音频分类模型取得了显著更优的性能，甚至在某些数据集上超越了音视频多模态分类模型。具体而言，我们的方法在AudioSet（2M、20K）和FSD50K等数据集上取得了突出成果，性能得分分别为53.9、45.1和65.6。我们已公开共享代码和模型：https://github.com/LZH-0225/AudioFormer.git。