Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information. However, current models often focus on a limited set of tasks, and generalization abilities of learned representations are unclear. To this end, we propose the AV-SUPERB benchmark that enables general-purpose evaluation of unimodal audio/visual and bimodal fusion representations on 7 datasets covering 5 audio-visual tasks in speech and audio processing. We evaluate 5 recent self-supervised models and show that none of these models generalize to all tasks, emphasizing the need for future study on improving universal model performance. In addition, we show that representations may be improved with intermediate-task fine-tuning and audio event classification with AudioSet serves as a strong intermediate task. We release our benchmark with evaluation code and a model submission platform to encourage further research in audio-visual learning.

翻译：音视频表征学习旨在通过利用听觉与视觉信息之间的关联性，开发具有类人感知能力的系统。然而，现有模型通常仅聚焦于有限的任务集，且所学表征的泛化能力尚不明确。为此，我们提出AV-SUPERB基准，该基准能够对单模态音频/视觉表征及双模态融合表征在覆盖语音与音频处理领域中5项音视频任务的7个数据集上进行通用评估。我们评估了5个近期自监督模型，结果表明这些模型均无法泛化至所有任务，这凸显了提升通用模型性能的未来研究需求。此外，我们验证了通过中间任务微调可提升表征质量，而基于AudioSet的音频事件分类可作为一项有效的中间任务。我们发布了包含评估代码和模型提交平台的基准，以推动音视频学习的进一步研究。