In this work, we address music representation learning using convolution-free transformers. We build on top of existing spectrogram-based audio transformers such as AST and train our models on a supervised task using patchout training similar to PaSST. In contrast to previous works, we study how specific design decisions affect downstream music tagging tasks instead of focusing on the training task. We assess the impact of initializing the models with different pre-trained weights, using various input audio segment lengths, using learned representations from different blocks and tokens of the transformer for downstream tasks, and applying patchout at inference to speed up feature extraction. We find that 1) initializing the model from ImageNet or AudioSet weights and using longer input segments are beneficial both for the training and downstream tasks, 2) the best representations for the considered downstream tasks are located in the middle blocks of the transformer, and 3) using patchout at inference allows faster processing than our convolutional baselines while maintaining superior performance. The resulting models, MAEST, are publicly available and obtain the best performance among open models in music tagging tasks.

翻译：在本工作中，我们探讨了利用无卷积Transformer进行音乐表示学习。我们基于现有频谱图音频Transformer（如AST）构建模型，并采用类似PaSST的补丁丢弃训练（patchout training）在监督任务上进行训练。与以往工作不同，我们专注于研究特定设计决策如何影响下游音乐标注任务，而非仅关注训练任务本身。我们评估了以下因素的影响：使用不同预训练权重初始化模型、采用不同输入音频片段长度、利用Transformer不同模块和令牌（token）的习得表示进行下游任务，以及在推理阶段应用补丁丢弃加速特征提取。研究结果表明：1）使用ImageNet或AudioSet权重初始化模型并采用更长的输入片段，对训练任务和下游任务均有益处；2）所考虑的下游任务的最佳表示位于Transformer的中间模块；3）推理时采用补丁丢弃可在保持优越性能的同时实现比卷积基线更快的处理速度。最终模型MAEST已公开，在音乐标注任务中取得了开放模型中的最优性能。