Never having seen an object and heard its sound simultaneously, can the model still accurately localize its visual position from the input audio? In this work, we concentrate on the Audio-Visual Localization and Segmentation tasks but under the demanding zero-shot and few-shot scenarios. To achieve this goal, different from existing approaches that mostly employ the encoder-fusion-decoder paradigm to decode localization information from the fused audio-visual feature, we introduce the encoder-prompt-decoder paradigm, aiming to better fit the data scarcity and varying data distribution dilemmas with the help of abundant knowledge from pre-trained models. Specifically, we first propose to construct Semantic-aware Audio Prompt (SAP) to help the visual foundation model focus on sounding objects, meanwhile, the semantic gap between the visual and audio modalities is also encouraged to shrink. Then, we develop a Correlation Adapter (ColA) to keep minimal training efforts as well as maintain adequate knowledge of the visual foundation model. By equipping with these means, extensive experiments demonstrate that this new paradigm outperforms other fusion-based methods in both the unseen class and cross-dataset settings. We hope that our work can further promote the generalization study of Audio-Visual Localization and Segmentation in practical application scenarios.

翻译：从未同时见过物体和听过其声音，模型能否仅凭输入音频准确定位其视觉位置？本文聚焦于视听定位与分割任务，但致力于应对苛刻的零样本和少样本场景。为此，我们摒弃现有方法主要采用的"编码器-融合-解码器"范式（即从融合的视听特征中解码定位信息），引入"编码器-提示-解码器"新范式，旨在借助预训练模型的丰富知识，更好地适配数据稀缺与数据分布多变的问题。具体而言，我们首先提出构建语义感知音频提示（Semantic-aware Audio Prompt, SAP），帮助视觉基础模型聚焦于发声物体，同时缩小视觉与音频模态之间的语义鸿沟。接着，我们开发了相关性适配器（Correlation Adapter, ColA），在保持视觉基础模型充足知识的同时，仅需极少的训练成本。通过配备这些手段，大量实验表明，该新范式在未见类别和跨数据集场景中均优于其他基于融合的方法。我们希望本工作能进一步推动视听定位与分割在现实应用场景中的泛化研究。