Self-supervised learning (SSL) has recently emerged as a promising paradigm for training generalisable models on large-scale data in the fields of vision, text, and speech. Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored. This is partially due to the distinctive challenges associated with modelling musical knowledge, particularly tonal and pitched characteristics of music. To address this research gap, we propose an acoustic Music undERstanding model with large-scale self-supervised Training (MERT), which incorporates teacher models to provide pseudo labels in the masked language modelling (MLM) style acoustic pre-training. In our exploration, we identified an effective combination of teacher models, which outperforms conventional speech and audio approaches in terms of performance. This combination includes an acoustic teacher based on Residual Vector Quantisation - Variational AutoEncoder (RVQ-VAE) and a musical teacher based on the Constant-Q Transform (CQT). Furthermore, we explore a wide range of settings to overcome the instability in acoustic language model pre-training, which allows our designed paradigm to scale from 95M to 330M parameters. Experimental results indicate that our model can generalise and perform well on 14 music understanding tasks and attain state-of-the-art (SOTA) overall scores.

翻译：自监督学习（SSL）近期在视觉、文本和语音领域的大规模数据训练中，已成为一种可实现通用化模型的有前景范式。尽管SSL在语音和音频任务中已证明其有效性，但其在音乐音频中的应用仍需深入探索。这 partly 源于音乐知识建模面临的独特挑战，尤其是音乐的音调与音高特征。为填补这一研究空白，我们提出了一种基于大规模自监督训练的声学音乐理解模型（MERT），该模型引入教师模型，在掩码语言建模（MLM）风格的声学预训练中提供伪标签。通过探索，我们确定了一种有效的教师模型组合，其性能优于传统语音和音频方法。该组合包含基于残差向量量化-变分自编码器（RVQ-VAE）的声学教师模型，以及基于常量Q变换（CQT）的音乐教师模型。此外，我们探索了多种设置以克服声学语言模型预训练中的不稳定性，从而使所设计的范式可扩展至从95M到330M的参数规模。实验结果表明，我们的模型在14项音乐理解任务上具有通用性且表现优异，并取得了整体最优（SOTA）得分。