Recently, audio-visual separation approaches have taken advantage of the natural synchronization between the two modalities to boost audio source separation performance. They extracted high-level semantics from visual inputs as the guidance to help disentangle sound representation for individual sources. Can we directly learn to disentangle the individual semantics from the sound itself? The dilemma is that multiple sound sources are mixed together in the original space. To tackle the difficulty, in this paper, we present a novel Semantic Grouping Network, termed as SGN, that can directly disentangle sound representations and extract high-level semantic information for each source from input audio mixture. Specifically, SGN aggregates category-wise source features through learnable class tokens of sounds. Then, the aggregated semantic features can be used as the guidance to separate the corresponding audio sources from the mixture. We conducted extensive experiments on music-only and universal sound separation benchmarks: MUSIC, FUSS, MUSDB18, and VGG-Sound. The results demonstrate that our SGN significantly outperforms previous audio-only methods and audio-visual models without utilizing additional visual cues.

翻译：近年来，音频-视觉分离方法利用两种模态间的自然同步性来提升音频源分离性能。这些方法从视觉输入中提取高层语义作为指导，以帮助解耦各个声源的音频表征。我们能否直接从音频本身学习解耦个体语义？其难点在于多个声源在原始空间中相互混合。为解决这一难题，本文提出一种新颖的语义分组网络（SGN），该网络能够直接从输入的混合音频中解耦音频表征，并为每个声源提取高层语义信息。具体而言，SGN通过可学习的音频类别标记聚合类别化声源特征。随后，聚合后的语义特征可作为指导从混合音频中分离对应声源。我们在纯音乐和通用声音分离基准数据集（MUSIC、FUSS、MUSDB18和VGG-Sound）上进行了大量实验。结果表明，在不使用额外视觉线索的情况下，我们的SGN显著优于先前仅使用音频的方法以及音频-视觉模型。