We propose a self-supervised learning approach for videos that learns representations of both the RGB frames and the accompanying audio without human supervision. In contrast to images that capture the static scene appearance, videos also contain sound and temporal scene dynamics. To leverage the temporal and aural dimension inherent to videos, our method extends temporal self-supervision to the audio-visual setting and integrates it with multi-modal contrastive objectives. As temporal self-supervision, we pose playback speed and direction recognition in both modalities and propose intra- and inter-modal temporal ordering tasks. Furthermore, we design a novel contrastive objective in which the usual pairs are supplemented with additional sample-dependent positives and negatives sampled from the evolving feature space. In our model, we apply such losses among video clips and between videos and their temporally corresponding audio clips. We verify our model design in extensive ablation experiments and evaluate the video and audio representations in transfer experiments to action recognition and retrieval on UCF101 and HMBD51, audio classification on ESC50, and robust video fingerprinting on VGG-Sound, with state-of-the-art results.

翻译：我们提出了一种面向视频的自监督学习方法，无需人工标注即可同时学习RGB帧序列及伴随音频的表示。与仅捕捉静态场景外观的图像不同，视频同时包含声音与动态场景时序信息。为充分利用视频固有的时序与听觉维度，本方法将时序自监督机制扩展至视听场景，并将其与多模态对比学习目标相结合。在时序自监督方面，我们针对两种模态设计了播放速度与方向识别任务，并提出模态内及跨模态时序排序任务。此外，我们构建了一种新型对比学习框架，在传统正负样本对基础上，从动态演化的特征空间中抽取额外的样本依赖正负样本。该损失函数同时应用于视频片段之间以及视频与其时序对应音频片段之间的对比学习。通过大量消融实验验证模型设计后，我们在UCF101和HMBD51数据集的动作识别与检索、ESC50数据集的声音分类以及VGG-Sound数据集的鲁棒视频指纹识别等迁移实验中评估了视频与音频表示性能，均取得了最先进的结果。