The excellent performance of recent self-supervised learning methods on various downstream tasks has attracted great attention from academia and industry. Some recent research efforts have been devoted to self-supervised music representation learning. Nevertheless, most of them learn to represent equally-sized music clips in the waveform or a spectrogram. Despite being effective in some tasks, learning music representations in such a manner largely neglect the inherent part-whole hierarchies of music. Due to the hierarchical nature of the auditory cortex [24], understanding the bottom-up structure of music, i.e., how different parts constitute the whole at different levels, is essential for music understanding and representation learning. This work pursues hierarchical music representation learning and introduces the Music-PAW framework, which enables feature interactions of cropped music clips with part-whole hierarchies. From a technical perspective, we propose a transformer-based part-whole interaction module to progressively reason the structural relationships between part-whole music clips at adjacent levels. Besides, to create a multi-hierarchy representation space, we devise a hierarchical contrastive learning objective to align part-whole music representations in adjacent hierarchies. The merits of audio representation learning from part-whole hierarchies have been validated on various downstream tasks, including music classification (single-label and multi-label), cover song identification and acoustic scene classification.

翻译：近期自监督学习方法在各类下游任务中的卓越表现引发了学术界和工业界的广泛关注。部分研究致力于自监督音乐表征学习，然而大多数方法仅对波形或频谱图中等长音乐片段进行表征。尽管此类方法在某些任务中有效，但严重忽视了音乐固有的部分-整体层级结构。由于听觉皮层具有层级特性[24]，理解音乐自底向上的结构（即不同部分如何在多个层级构成整体）对音乐理解与表征学习至关重要。本研究探索层级化音乐表征学习，提出Music-PAW框架，该框架能够实现裁剪音乐片段在部分-整体层级结构中的特征交互。技术层面，我们提出基于Transformer的部分-整体交互模块，渐进推断相邻层级部分-整体音乐片段间的结构关联。此外，为构建多层级表征空间，我们设计层级化对比学习目标函数，对齐相邻层级中的部分-整体音乐表征。在音乐分类（单标签与多标签）、翻唱识别及声学场景分类等多项下游任务中，验证了从部分-整体层级进行音频表征学习的优势。