Self-supervised learning (SSL) has recently emerged as a promising paradigm for training generalisable models on large-scale data in the fields of vision, text, and speech. Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored. This is primarily due to the distinctive challenges associated with modelling musical knowledge, particularly its tonal and pitched characteristics of music. To address this research gap, we propose an acoustic Music undERstanding model with large-scale self-supervised Training (MERT), which incorporates teacher models to provide pseudo labels in the masked language modelling (MLM) style acoustic pre-training. In our exploration, we identified a superior combination of teacher models, which outperforms conventional speech and audio approaches in terms of performance. This combination includes an acoustic teacher based on Residual Vector Quantization - Variational AutoEncoder (RVQ-VAE) and a musical teacher based on the Constant-Q Transform (CQT). These teachers effectively guide our student model, a BERT-style transformer encoder, to better model music audio. In addition, we introduce an in-batch noise mixture augmentation to enhance the representation robustness. Furthermore, we explore a wide range of settings to overcome the instability in acoustic language model pre-training, which allows our designed paradigm to scale from 95M to 330M parameters. Experimental results indicate that our model can generalise and perform well on 14 music understanding tasks and attains state-of-the-art (SOTA) overall scores. The code and models are online: https://github.com/yizhilll/MERT.

翻译：自监督学习（SSL）近年来在视觉、文本和语音领域的大规模数据通用模型训练中展现出良好前景。尽管SSL在语音和音频领域已被证明有效，但其在音乐音频中的应用尚未得到充分探索。这主要源于音乐知识建模的特殊挑战，特别是其调性和音高特征。为填补这一研究空白，我们提出了一种基于大规模自监督训练的声学音乐理解模型（MERT），该模型引入教师模型，以掩码语言建模（MLM）风格为声学预训练提供伪标签。通过探索，我们确定了优于传统语音与音频方法的教师模型组合方案，包括基于残差向量量化-变分自编码器（RVQ-VAE）的声学教师模型和基于常数Q变换（CQT）的音乐教师模型。这些教师模型有效引导我们的学生模型（采用BERT风格的Transformer编码器）更好地建模音乐音频。此外，我们引入批内噪声混合增强方法以提升表示鲁棒性，并广泛探索了多种设置以克服声学语言模型预训练中的不稳定性，使得所设计范式可扩展至从9500万到3.3亿参数。实验结果表明，我们的模型在14项音乐理解任务中表现优异，整体得分达到当前最优水平（SOTA）。代码与模型已开源：https://github.com/yizhilll/MERT.