Tracking by natural language specification aims to locate the referred target in a sequence based on the natural language description. Existing algorithms solve this issue in two steps, visual grounding and tracking, and accordingly deploy the separated grounding model and tracking model to implement these two steps, respectively. Such a separated framework overlooks the link between visual grounding and tracking, which is that the natural language descriptions provide global semantic cues for localizing the target for both two steps. Besides, the separated framework can hardly be trained end-to-end. To handle these issues, we propose a joint visual grounding and tracking framework, which reformulates grounding and tracking as a unified task: localizing the referred target based on the given visual-language references. Specifically, we propose a multi-source relation modeling module to effectively build the relation between the visual-language references and the test image. In addition, we design a temporal modeling module to provide a temporal clue with the guidance of the global semantic information for our model, which effectively improves the adaptability to the appearance variations of the target. Extensive experimental results on TNL2K, LaSOT, OTB99, and RefCOCOg demonstrate that our method performs favorably against state-of-the-art algorithms for both tracking and grounding. Code is available at https://github.com/lizhou-cs/JointNLT.

翻译：基于自然语言规范的跟踪旨在根据自然语言描述在序列中定位所指目标。现有算法通常分两步解决该问题：视觉定位与跟踪，并相应部署独立的定位模型和跟踪模型分别执行这两个步骤。这种分离框架忽视了视觉定位与跟踪之间的关联——自然语言描述为这两步的目标定位提供了全局语义线索。此外，分离框架难以实现端到端训练。针对这些问题，我们提出了一种联合视觉定位与跟踪框架，将定位与跟踪重构为统一任务：基于给定的视觉-语言参照来定位所指目标。具体而言，我们设计了一个多源关系建模模块，有效构建视觉-语言参照与测试图像之间的关系。同时，引入时序建模模块，在全局语义信息引导下为模型提供时序线索，显著提升了对目标外观变化的适应性。在TNL2K、LaSOT、OTB99和RefCOCOg数据集上的大量实验表明，该方法在跟踪与定位任务上均优于现有最优算法。代码已开源至 https://github.com/lizhou-cs/JointNLT。