Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information. However, current models often focus on a limited set of tasks, and generalization abilities of learned representations are unclear. To this end, we propose the AV-SUPERB benchmark that enables general-purpose evaluation of unimodal audio/visual and bimodal fusion representations on 7 datasets covering 5 audio-visual tasks in speech and audio processing. We evaluate 5 recent self-supervised models and show that none of these models generalize to all tasks, emphasizing the need for future study on improving universal model performance. In addition, we show that representations may be improved with intermediate-task fine-tuning and audio event classification with AudioSet serves as a strong intermediate task. We release our benchmark with evaluation code and a model submission platform to encourage further research in audio-visual learning.

翻译：音视频表示学习旨在通过利用听觉与视觉信息之间的相关性，开发具有类人感知能力的系统。然而，当前模型常局限于有限的任务集，所学表示的泛化能力尚不明确。为此，我们提出AV-SUPERB基准，该基准可在覆盖语音与音频处理中5项音视频任务的7个数据集上，对单模态音频/视觉表示及双模态融合表示进行通用评估。我们评估了5种近期自监督模型，结果显示无任何模型可泛化至所有任务，这凸显了未来在提升通用模型性能方面的研究需求。此外，研究表明可通过中间任务微调改进表示，而基于AudioSet的音频事件分类可充当有效的中间任务。我们发布包含评估代码与模型提交平台的基准，以推动音视频学习的进一步研究。