Multitrack music transcription aims to transcribe a music audio input into the musical notes of multiple instruments simultaneously. It is a very challenging task that typically requires a more complex model to achieve satisfactory result. In addition, prior works mostly focus on transcriptions of regular instruments, however, neglecting vocals, which are usually the most important signal source if present in a piece of music. In this paper, we propose a novel deep neural network architecture, Perceiver TF, to model the time-frequency representation of audio input for multitrack transcription. Perceiver TF augments the Perceiver architecture by introducing a hierarchical expansion with an additional Transformer layer to model temporal coherence. Accordingly, our model inherits the benefits of Perceiver that posses better scalability, allowing it to well handle transcriptions of many instruments in a single model. In experiments, we train a Perceiver TF to model 12 instrument classes as well as vocal in a multi-task learning manner. Our result demonstrates that the proposed system outperforms the state-of-the-art counterparts (e.g., MT3 and SpecTNT) on various public datasets.

翻译：多轨音乐转录旨在将音乐音频输入同步转录为多种乐器的音符。这是一项极具挑战性的任务，通常需要更复杂的模型才能获得满意结果。此外，现有研究大多聚焦于常规乐器的转录，却忽略了人声——这在音乐作品中往往是最重要的信号源。本文提出一种新型深度神经网络架构——Perceiver TF，用于建模音频输入的时频特征以实现多轨转录。Perceiver TF通过引入层次化扩展，在原有Perceiver架构上增加额外的Transformer层以建模时间连贯性。由此，我们的模型继承了Perceiver的扩展性优势，使其能够通过单一模型有效处理多种乐器的转录任务。实验中，我们以多任务学习方式训练Perceiver TF模型，涵盖12个乐器类别及人声。结果表明，所提系统在多个公开数据集上均优于当前最优方法（如MT3和SpecTNT）。