The ability to recognize, localize and track dynamic objects in a scene is fundamental to many real-world applications, such as self-driving and robotic systems. Yet, traditional multiple object tracking (MOT) benchmarks rely only on a few object categories that hardly represent the multitude of possible objects that are encountered in the real world. This leaves contemporary MOT methods limited to a small set of pre-defined object categories. In this paper, we address this limitation by tackling a novel task, open-vocabulary MOT, that aims to evaluate tracking beyond pre-defined training categories. We further develop OVTrack, an open-vocabulary tracker that is capable of tracking arbitrary object classes. Its design is based on two key ingredients: First, leveraging vision-language models for both classification and association via knowledge distillation; second, a data hallucination strategy for robust appearance feature learning from denoising diffusion probabilistic models. The result is an extremely data-efficient open-vocabulary tracker that sets a new state-of-the-art on the large-scale, large-vocabulary TAO benchmark, while being trained solely on static images. Project page: https://www.vis.xyz/pub/ovtrack/

翻译：场景中动态物体的识别、定位与追踪能力是自动驾驶和机器人系统等众多实际应用的核心基础。然而，传统的多目标跟踪（MOT）基准仅依赖少数目标类别，难以涵盖现实世界中可能遇到的各种物体。这导致当前的MOT方法局限于少量预定义目标类别。本文针对这一局限，提出一项新任务——开放词汇多目标追踪（open-vocabulary MOT），旨在评估超越预定义训练类别的追踪能力。我们进一步开发了OVTrack，一种能够追踪任意目标类别的开放词汇追踪器。其设计基于两个关键要素：首先，利用视觉-语言模型通过知识蒸馏实现分类与关联；其次，采用数据幻觉策略，利用去噪扩散概率模型学习鲁棒的外观特征。最终，我们仅使用静态图像训练，便获得了极其数据高效的开放词汇追踪器，在大规模、大词汇量的TAO基准上创造了新纪录。项目主页：https://www.vis.xyz/pub/ovtrack/