The goal of the audio-visual segmentation (AVS) task is to segment the sounding objects in the video frames using audio cues. However, current fusion-based methods have the performance limitations due to the small receptive field of convolution and inadequate fusion of audio-visual features. To overcome these issues, we propose a novel \textbf{Au}dio-aware query-enhanced \textbf{TR}ansformer (AuTR) to tackle the task. Unlike existing methods, our approach introduces a multimodal transformer architecture that enables deep fusion and aggregation of audio-visual features. Furthermore, we devise an audio-aware query-enhanced transformer decoder that explicitly helps the model focus on the segmentation of the pinpointed sounding objects based on audio signals, while disregarding silent yet salient objects. Experimental results show that our method outperforms previous methods and demonstrates better generalization ability in multi-sound and open-set scenarios.

翻译：视听分割（AVS）任务的目标是利用音频线索对视频帧中的发声物体进行分割。然而，当前基于融合的方法由于卷积感受野较小以及视听特征融合不充分，存在性能局限性。为解决这些问题，我们提出了一种新颖的音频感知查询增强Transformer（AuTR）来处理该任务。与现有方法不同，我们的方法引入了一种多模态Transformer架构，能够实现视听特征的深度融合与聚合。此外，我们设计了一个音频感知查询增强Transformer解码器，该解码器能显式帮助模型基于音频信号专注于精准定位的发声物体分割，同时忽略无声但显著的物体。实验结果表明，我们的方法优于以往方法，并在多声源和开放场景下展现出更强的泛化能力。