Semantic Multi-Object Tracking (SMOT) extends multi-object tracking with semantic outputs such as video summaries, instance-level captions, and interaction labels, aiming to move from trajectories to human-interpretable descriptions of dynamic scenes. Existing SMOT systems are trained end-to-end, coupling progress to expensive supervision, limiting the ability to rapidly adapt to new foundation models and new interactions. We propose TF-SMOT, a training-free SMOT pipeline that composes pretrained components for detection, mask-based tracking, and video-language generation. TF-SMOT combines D-FINE and the promptable SAM2 segmentation tracker to produce temporally consistent tracklets, uses contour grounding to generate video summaries and instance captions with InternVideo2.5, and aligns extracted interaction predicates to BenSMOT WordNet synsets via gloss-based semantic retrieval with LLM disambiguation. On BenSMOT, TF-SMOT achieves state-of-the-art tracking performance within the SMOT setting and improves summary and caption quality compared to prior art. Interaction recognition, however, remains challenging under strict exact-match evaluation on the fine-grained and long-tailed WordNet label space; our analysis and ablations indicate that semantic overlap and label granularity substantially affect measured performance.

翻译：语义多目标跟踪（SMOT）在传统多目标跟踪基础上引入语义输出，例如视频摘要、实例级描述和交互标签，旨在将轨迹转换为人可理解的动态场景描述。现有SMOT系统采用端到端训练方式，这一过程依赖于昂贵的监督数据，限制了其快速适配新基础模型和新交互行为的能力。本文提出TF-SMOT，一种无需训练的SMOT流水线，通过组合预训练组件实现检测、基于掩码的跟踪以及视频-语言生成。TF-SMOT结合D-FINE与可提示的SAM2分割跟踪器生成时序一致的短轨迹，利用轮廓定位技术通过InternVideo2.5生成视频摘要和实例描述，并基于带LLM消歧的词汇语义检索将提取的交互谓词对齐至BenSMOT WordNet同义词集。在BenSMOT基准上，TF-SMOT在SMOT设置下取得了最先进的跟踪性能，并在摘要和描述质量上超越现有技术。然而，在细粒度且长尾的WordNet标签空间严格精确匹配评估下，交互识别仍具挑战性；我们的分析与消融实验表明，语义重叠和标签粒度对测量性能有显著影响。