This paper introduces the task of Auditory Referring Multi-Object Tracking (AR-MOT), which dynamically tracks specific objects in a video sequence based on audio expressions and appears as a challenging problem in autonomous driving. Due to the lack of semantic modeling capacity in audio and video, existing works have mainly focused on text-based multi-object tracking, which often comes at the cost of tracking quality, interaction efficiency, and even the safety of assistance systems, limiting the application of such methods in autonomous driving. In this paper, we delve into the problem of AR-MOT from the perspective of audio-video fusion and audio-video tracking. We put forward EchoTrack, an end-to-end AR-MOT framework with dual-stream vision transformers. The dual streams are intertwined with our Bidirectional Frequency-domain Cross-attention Fusion Module (Bi-FCFM), which bidirectionally fuses audio and video features from both frequency- and spatiotemporal domains. Moreover, we propose the Audio-visual Contrastive Tracking Learning (ACTL) regime to extract homogeneous semantic features between expressions and visual objects by learning homogeneous features between different audio and video objects effectively. Aside from the architectural design, we establish the first set of large-scale AR-MOT benchmarks, including Echo-KITTI, Echo-KITTI+, and Echo-BDD. Extensive experiments on the established benchmarks demonstrate the effectiveness of the proposed EchoTrack model and its components. The source code and datasets will be made publicly available at https://github.com/lab206/EchoTrack.

翻译：本文提出了听觉指代多目标跟踪（AR-MOT）任务，该任务基于音频表达动态跟踪视频序列中的特定目标，是自动驾驶领域的一个挑战性问题。由于音频和视频缺乏语义建模能力，现有工作主要集中于文本驱动的多目标跟踪，这往往以牺牲跟踪质量、交互效率乃至辅助系统安全性为代价，限制了此类方法在自动驾驶中的应用。本文从音视频融合与音视频联合跟踪的角度深入探究了AR-MOT问题。我们提出了EchoTrack——一种端到端的AR-MOT框架，采用双流视觉Transformer架构。双流通过双向频域交叉注意力融合模块（Bi-FCFM）交织，该模块从频域和时空域双向融合音频与视频特征。此外，我们提出音视频对比跟踪学习（ACTL）机制，通过有效学习不同音视频目标之间的同质特征，提取表达式与视觉目标之间的同质语义特征。除架构设计外，我们首次建立了大规模AR-MOT基准数据集，包括Echo-KITTI、Echo-KITTI+和Echo-BDD。在已构建基准上的大量实验证明了所提出的EchoTrack模型及其各组件的有效性。源代码与数据集将开源发布于https://github.com/lab206/EchoTrack。