Most existing masked audio modeling (MAM) methods learn audio representations by masking and reconstructing local spectrogram patches. However, the reconstruction loss mainly accounts for the signal-level quality of the reconstructed spectrogram and is still limited in extracting high-level audio semantics. In this paper, we propose to enhance the semantic modeling of MAM by distilling cross-modality knowledge from contrastive language-audio pretraining (CLAP) representations for both masked and unmasked regions (MAM-CLAP) and leveraging a multi-objective learning strategy with a supervised classification branch (SupMAM), thereby providing more semantic knowledge for MAM and enabling it to effectively learn global features from labels. Experiments show that our methods significantly improve the performance on multiple downstream tasks. Furthermore, by combining our MAM-CLAP with SupMAM, we can achieve new state-of-the-art results on various audio and speech classification tasks, exceeding previous self-supervised learning and supervised pretraining methods.

翻译：现有的掩码音频建模（MAM）方法大多通过对局部频谱图块进行掩码和重构来学习音频表示。然而，重构损失主要侧重于重构频谱图的信号级质量，在提取高层音频语义方面仍存在局限。本文提出通过从对比语言-音频预训练（CLAP）表示中蒸馏跨模态知识，并应用于掩码和非掩码区域（MAM-CLAP），同时结合有监督分类分支（SupMAM）的多目标学习策略，从而为MAM提供更多语义知识，使其能够从标签中有效学习全局特征。实验表明，我们的方法显著提升了多个下游任务的性能。此外，通过将MAM-CLAP与SupMAM相结合，我们在多种音频和语音分类任务中取得了新的最佳结果，超越了以往的自监督学习和有监督预训练方法。