In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities. Subsequently, we propose the Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) by combining contrastive learning and masked data modeling, two major self-supervised learning frameworks, to learn a joint and coordinated audio-visual representation. Our experiments show that the contrastive audio-visual correspondence learning objective not only enables the model to perform audio-visual retrieval tasks, but also helps the model learn a better joint representation. As a result, our fully self-supervised pretrained CAV-MAE achieves a new SOTA accuracy of 65.9% on VGGSound, and is comparable with the previous best supervised pretrained model on AudioSet in the audio-visual event classification task. Code and pretrained models are at https://github.com/yuangongnd/cav-mae.

翻译：本文首先将近期提出的掩码自编码器（MAE）模型从单一模态扩展到视听多模态。随后，我们结合对比学习与掩码数据建模这两种主要的自监督学习框架，提出对比视听掩码自编码器（CAV-MAE），以学习联合协调的视听表征。实验表明，对比视听对应学习目标不仅使模型能够执行视听检索任务，还有助于模型学习更优的联合表征。最终，我们的完全自监督预训练CAV-MAE在VGGSound数据集上取得了65.9%的新SOTA准确率，并在视听事件分类任务中与AudioSet上此前最优的有监督预训练模型性能相当。代码与预训练模型已开源至https://github.com/yuangongnd/cav-mae。