Existing video instance segmentation (VIS) approaches generally follow a closed-world assumption, where only seen category instances are identified and spatio-temporally segmented at inference. Open-world formulation relaxes the close-world static-learning assumption as follows: (a) first, it distinguishes a set of known categories as well as labels an unknown object as `unknown' and then (b) it incrementally learns the class of an unknown as and when the corresponding semantic labels become available. We propose the first open-world VIS approach, named OW-VISFormer, that introduces a novel feature enrichment mechanism and a spatio-temporal objectness (STO) module. The feature enrichment mechanism based on a light-weight auxiliary network aims at accurate pixel-level (unknown) object delineation from the background as well as distinguishing category-specific known semantic classes. The STO module strives to generate instance-level pseudo-labels by enhancing the foreground activations through a contrastive loss. Moreover, we also introduce an extensive experimental protocol to measure the characteristics of OW-VIS. Our OW-VISFormer performs favorably against a solid baseline in OW-VIS setting. Further, we evaluate our contributions in the standard fully-supervised VIS setting by integrating them into the recent SeqFormer, achieving an absolute gain of 1.6\% AP on Youtube-VIS 2019 val. set. Lastly, we show the generalizability of our contributions for the open-world detection (OWOD) setting, outperforming the best existing OWOD method in the literature. Code, models along with OW-VIS splits are available at \url{https://github.com/OmkarThawakar/OWVISFormer}.

翻译：现有的视频实例分割（VIS）方法通常遵循封闭世界假设，即推理时仅识别并时空分割已知类别的实例。开放世界范式通过以下方式放松了封闭世界的静态学习假设：（a）首先区分一组已知类别，并将未知物体标记为“未知”，然后（b）随着对应语义标签的出现，逐步学习未知物体的类别。我们提出了首个开放世界VIS方法，命名为OW-VISFormer，该方法引入了新颖的特征增强机制和时空目标性（STO）模块。基于轻量级辅助网络的特征增强机制旨在从背景中精确描绘像素级（未知）物体，并区分类别特定的已知语义类。STO模块通过对比学习增强前景激活，努力生成实例级伪标签。此外，我们还引入了一套广泛的实验协议来衡量OW-VIS的特性。在OW-VIS设置下，我们的OW-VISFormer相较于强基线表现优异。进一步地，我们通过将贡献集成到近期SeqFormer中，在标准全监督VIS设置下评估其效果，在YouTube-VIS 2019验证集上实现了1.6% AP的绝对提升。最后，我们展示了贡献在开放世界检测（OWOD）设置中的泛化能力，超越了现有文献中最优的OWOD方法。代码、模型及OW-VIS划分详见\url{https://github.com/OmkarThawakar/OWVISFormer}。