In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities. Subsequently, we propose the Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) by combining contrastive learning and masked data modeling, two major self-supervised learning frameworks, to learn a joint and coordinated audio-visual representation. Our experiments show that the contrastive audio-visual correspondence learning objective not only enables the model to perform audio-visual retrieval tasks, but also helps the model learn a better joint representation. As a result, our fully self-supervised pretrained CAV-MAE achieves a new SOTA accuracy of 65.9% on VGGSound, and is comparable with the previous best supervised pretrained model on AudioSet in the audio-visual event classification task.

翻译：在本文中,我们首先将最近的蒙面自动编码器模式从单一模式扩大到视听多模式。 随后,我们提出将对比性视听蒙面自动编码器模式(CAV-MAE)结合对比性学习和蒙面数据模型(两个主要自我监督的学习框架)来学习联合和协调的视听演示。 我们的实验表明,对比性视听函授学习目标不仅使模型能够执行视听检索任务,而且有助于模型学习更好的联合代表。 结果,我们完全由自我监督的事先经过培训的CAVMAE在VGSound上实现了65.9%的新的SOTA精度,并且与先前在视听事件分类任务中受监督的关于音频卫星的预先培训模式相当。