Recently, an audio-visual segmentation (AVS) task has been introduced, aiming to group pixels with sounding objects within a given video. This task necessitates a first-ever audio-driven pixel-level understanding of the scene, posing significant challenges. In this paper, we propose an innovative audio-visual transformer framework, termed COMBO, an acronym for COoperation of Multi-order Bilateral relatiOns. For the first time, our framework explores three types of bilateral entanglements within AVS: pixel entanglement, modality entanglement, and temporal entanglement. Regarding pixel entanglement, we employ a Siam-Encoder Module (SEM) that leverages prior knowledge to generate more precise visual features from the foundational model. For modality entanglement, we design a Bilateral-Fusion Module (BFM), enabling COMBO to align corresponding visual and auditory signals bi-directionally. As for temporal entanglement, we introduce an innovative adaptive inter-frame consistency loss according to the inherent rules of temporal. Comprehensive experiments and ablation studies on AVSBench-object (84.7 mIoU on S4, 59.2 mIou on MS3) and AVSBench-semantic (42.1 mIoU on AVSS) datasets demonstrate that COMBO surpasses previous state-of-the-art methods. Code and more results will be publicly available at https://combo-avs.github.io/.

翻译：最近，音视频分割（AVS）任务被提出，旨在对给定视频中发出声音的对象进行像素级分组。该任务首次需要对场景进行音频驱动的像素级理解，带来了巨大挑战。本文提出了一种创新的音视频Transformer框架，称为COMBO（即多阶双边关系协作的缩写）。我们的框架首次探索了AVS中的三种双边纠缠：像素纠缠、模态纠缠和时间纠缠。针对像素纠缠，我们采用Siam编码模块（SEM）利用先验知识从基础模型中生成更精确的视觉特征。针对模态纠缠，我们设计了双边融合模块（BFM），使COMBO能够双向对齐对应的视觉和听觉信号。针对时间纠缠，我们根据时间的内在规律引入了一种创新的自适应帧间一致性损失。在AVSBench-object（S4上84.7 mIoU，MS3上59.2 mIoU）和AVSBench-semantic（AVSS上42.1 mIoU）数据集上的全面实验和消融研究表明，COMBO超越了以往的最先进方法。代码及更多结果将在https://combo-avs.github.io/上公开。