We propose a method named AudioFormer, which learns audio feature representations through the acquisition of discrete acoustic codes and subsequently fine-tunes them for audio classification tasks. Initially, we introduce a novel perspective by considering the audio classification task as a form of natural language understanding (NLU). Leveraging an existing neural audio codec model, we generate discrete acoustic codes and utilize them to train a masked language model (MLM), thereby obtaining audio feature representations. Furthermore, we pioneer the integration of a \textbf{M}ulti-\textbf{P}ositive sample \textbf{C}ontrastive (MPC) learning approach. This method enables the learning of joint representations among multiple discrete acoustic codes within the same audio input. In our experiments, we treat discrete acoustic codes as textual data and train a masked language model using a cloze-like methodology, ultimately deriving high-quality audio representations. Notably, the MPC learning technique effectively captures collaborative representations among distinct positive samples. Our research outcomes demonstrate that AudioFormer attains significantly improved performance compared to prevailing monomodal audio classification models across multiple datasets, and even outperforms audio-visual multimodal classification models on select datasets. Specifically, our approach achieves remarkable results on datasets including AudioSet (2M, 20K), and FSD50K, with performance scores of 53.9, 45.1, and 65.6, respectively. We have openly shared both the code and models: \url{https://github.com/LZH-0225/AudioFormer.git}.

翻译：我们提出一种名为AudioFormer的方法，该方法通过获取离散声学编码学习音频特征表示，并随后对其进行微调以用于音频分类任务。首先，我们引入一个新视角，将音频分类任务视为一种自然语言理解（NLU）形式。利用现有的神经音频编解码器模型，我们生成离散声学编码，并利用它们训练掩码语言模型（MLM），从而获得音频特征表示。此外，我们率先整合了一种\textbf{多}正样本\textbf{对比}学习（MPC）方法。该方法能够学习同一音频输入中多个离散声学编码之间的联合表示。在我们的实验中，我们将离散声学编码视为文本数据，采用类似完形填空的方法训练掩码语言模型，最终推导出高质量的音频表示。值得注意的是，MPC学习技术有效捕捉了不同正样本之间的协作表示。我们的研究结果表明，AudioFormer在多个数据集上的性能显著优于当前主流的单模态音频分类模型，甚至在某些数据集上超越音频-视觉多模态分类模型。具体而言，我们的方法在AudioSet（2M, 20K）和FSD50K等数据集上取得了卓越结果，性能得分分别为53.9、45.1和65.6。我们已公开共享代码和模型：\url{https://github.com/LZH-0225/AudioFormer.git}。