Audio-visual segmentation (AVS) is a complex task that involves accurately segmenting the corresponding sounding object based on audio-visual queries. Successful audio-visual learning requires two essential components: 1) an unbiased dataset with high-quality pixel-level multi-class labels, and 2) a model capable of effectively linking audio information with its corresponding visual object. However, these two requirements are only partially addressed by current methods, with training sets containing biased audio-visual data, and models that generalise poorly beyond this biased training set. In this work, we propose a new strategy to build cost-effective and relatively unbiased audio-visual semantic segmentation benchmarks. Our strategy, called Visual Post-production (VPO), explores the observation that it is not necessary to have explicit audio-visual pairs extracted from single video sources to build such benchmarks. We also refine the previously proposed AVSBench to transform it into the audio-visual semantic segmentation benchmark AVSBench-Single+. Furthermore, this paper introduces a new pixel-wise audio-visual contrastive learning method to enable a better generalisation of the model beyond the training set. We verify the validity of the VPO strategy by showing that state-of-the-art (SOTA) models trained with datasets built by matching audio and visual data from different sources or with datasets containing audio and visual data from the same video source produce almost the same accuracy. Then, using the proposed VPO benchmarks and AVSBench-Single+, we show that our method produces more accurate audio-visual semantic segmentation than SOTA models. Code and dataset will be available.

翻译：音频-视觉分割（AVS）是一项复杂任务，涉及基于音频-视觉查询精确分割对应的发声物体。成功的音频-视觉学习需要两个关键组成部分：1）一个包含高质量像素级多类别标签的无偏数据集，以及2）一个能有效将音频信息与其对应视觉目标关联起来的模型。然而，当前方法仅部分满足了这两个需求，训练集包含有偏的音频-视觉数据，且模型在此有偏训练集之外的泛化能力较差。本文提出了一种构建经济高效且相对无偏的音频-视觉语义分割基准的新策略。该策略名为视觉后期制作（VPO），其依据是构建此类基准并不需要从单一视频源提取显式的音频-视觉对。同时，我们对先前提出的AVSBench进行改进，将其转化为音频-视觉语义分割基准AVSBench-Single+。此外，本文引入了一种新的逐像素音频-视觉对比学习方法，以提升模型在训练集之外的泛化能力。我们验证了VPO策略的有效性，结果表明，使用由不同来源的音频和视觉数据匹配构建的数据集训练的最先进（SOTA）模型，与使用同一视频源的音频和视觉数据构建的数据集训练的模型，准确率几乎相同。随后，利用提出的VPO基准和AVSBench-Single+，我们证明所提出的方法比SOTA模型能实现更精确的音频-视觉语义分割。代码和数据集将公开提供。