Tracking any given object(s) spatially and temporally is a common purpose in Visual Object Tracking (VOT) and Video Object Segmentation (VOS). Joint tracking and segmentation have been attempted in some studies but they often lack full compatibility of both box and mask in initialization and prediction, and mainly focus on single-object scenarios. To address these limitations, this paper proposes a Multi-object Mask-box Integrated framework for unified Tracking and Segmentation, dubbed MITS. Firstly, the unified identification module is proposed to support both box and mask reference for initialization, where detailed object information is inferred from boxes or directly retained from masks. Additionally, a novel pinpoint box predictor is proposed for accurate multi-object box prediction, facilitating target-oriented representation learning. All target objects are processed simultaneously from encoding to propagation and decoding, as a unified pipeline for VOT and VOS. Experimental results show MITS achieves state-of-the-art performance on both VOT and VOS benchmarks. Notably, MITS surpasses the best prior VOT competitor by around 6% on the GOT-10k test set, and significantly improves the performance of box initialization on VOS benchmarks. The code is available at https://github.com/yoxu515/MITS.

翻译：在时空维度上跟踪任意给定对象是视觉目标跟踪（VOT）和视频对象分割（VOS）的常见目标。已有研究尝试将跟踪与分割联合处理，但往往在初始化和预测阶段缺乏框与掩码的完全兼容性，且主要聚焦于单目标场景。为克服这些局限，本文提出一种多目标掩码-框集成框架用于统一跟踪与分割，简称MITS。首先，提出统一标识模块，支持通过框或掩码参考进行初始化，从而从框中推断或从掩码中直接保留详细对象信息。此外，设计了一种新颖的 pinpoint 框预测器，用于精确的多目标框预测，促进面向目标的表示学习。所有目标对象从编码到传播再到解码均同步处理，形成VOT和VOS的统一流水线。实验结果表明，MITS在VOT和VOS基准测试中均达到最先进性能。值得注意的是，在GOT-10k测试集上，MITS较之前最优VOT方法提升约6%，并在VOS基准测试中显著提升了框初始化的性能。代码开源在https://github.com/yoxu515/MITS。