One of the recent trends in vision problems is to use natural language captions to describe the objects of interest. This approach can overcome some limitations of traditional methods that rely on bounding boxes or category annotations. This paper introduces a novel paradigm for Multiple Object Tracking called Type-to-Track, which allows users to track objects in videos by typing natural language descriptions. We present a new dataset for that Grounded Multiple Object Tracking task, called GroOT, that contains videos with various types of objects and their corresponding textual captions describing their appearance and action in detail. Additionally, we introduce two new evaluation protocols and formulate evaluation metrics specifically for this task. We develop a new efficient method that models a transformer-based eMbed-ENcoDE-extRact framework (MENDER) using the third-order tensor decomposition. The experiments in five scenarios show that our MENDER approach outperforms another two-stage design in terms of accuracy and efficiency, up to 14.7% accuracy and 4$\times$ speed faster.

翻译：近期视觉问题研究的一个趋势是利用自然语言描述来标注感兴趣的对象。这种方法能够克服传统方法依赖边界框或类别标注的局限性。本文提出了一种名为Type-to-Track的多目标跟踪新范式，使用户能够通过输入自然语言描述来跟踪视频中的对象。我们针对这种基于文本的多目标跟踪任务构建了一个新数据集GroOT，其中包含各类对象的视频及其详细描述外观与动作的文本标注。此外，我们为该任务设计了两种新的评估协议并制定了专用评价指标。我们开发了一种基于三阶张量分解的高效Transformer框架（MENDER），该框架采用嵌入-编码-提取架构。在五个场景下的实验表明，我们的MENDER方法在准确率和效率上均优于另一种两阶段设计方法，准确率提升达14.7%，速度提高4倍。