We present \ourmodel{}, a simple Open-vocabulary Segmentation and Detection framework that jointly learns from different segmentation and detection datasets. To bridge the gap of vocabulary and annotation granularity, we first introduce a pre-trained text encoder to encode all the visual concepts in two tasks and learn a common semantic space for them. This gives us reasonably good results compared with the counterparts trained on segmentation task only. To further reconcile them, we locate two discrepancies: $i$) task discrepancy -- segmentation requires extracting masks for both foreground objects and background stuff, while detection merely cares about the former; $ii$) data discrepancy -- box and mask annotations are with different spatial granularity, and thus not directly interchangeable. To address these issues, we propose a decoupled decoding to reduce the interference between foreground/background and a conditioned mask decoding to assist in generating masks for given boxes. To this end, we develop a simple encoder-decoder model encompassing all three techniques and train it jointly on COCO and Objects365. After pre-training, our model exhibits competitive or stronger zero-shot transferability for both segmentation and detection. Specifically, \ourmodel{} beats the state-of-the-art method for open-vocabulary instance and panoptic segmentation across 5 datasets, and outperforms previous work for open-vocabulary detection on LVIS and ODinW under similar settings. When transferred to specific tasks, our model achieves new SoTA for panoptic segmentation on COCO and ADE20K, and instance segmentation on ADE20K and Cityscapes. Finally, we note that \ourmodel{} is the first to explore the potential of joint training on segmentation and detection, and hope it can be received as a strong baseline for developing a single model for both tasks in open world.

翻译：我们提出\ourmodel{}，这是一个简单的开放词汇分割与检测框架，能够联合学习来自不同分割与检测数据集的知识。为弥合词汇与标注粒度的差异，我们首先引入预训练文本编码器，对两项任务中的所有视觉概念进行编码，并为其学习一个共同语义空间。与仅使用分割任务训练的同类方法相比，该方法已获得相当不错的结果。为进一步协调两项任务，我们定位了两个差异点：$i$)任务差异——分割需同时提取前景对象与背景材质的掩码，而检测仅关注前者；$ii$)数据差异——边界框与掩码标注具有不同的空间粒度，因此无法直接互换。针对这些问题，我们提出解耦式解码以减少前景/背景间的相互干扰，以及条件掩码解码辅助为给定边界框生成掩码。由此，我们开发了一个融合上述三种技术的简单编码器-解码器模型，并在COCO与Objects365数据集上联合训练。预训练后，我们的模型在分割与检测任务上均展现出具有竞争力或更强的零样本迁移能力。具体而言，\ourmodel{}在5个数据集上超越了开放词汇实例分割与全景分割的现有最优方法；在相似设定下，于LVIS和ODinW数据集上的开放词汇检测任务中也优于先前工作。当迁移至特定任务时，我们的模型在COCO与ADE20K数据集的全景分割，以及ADE20K与Cityscapes数据集的实例分割上均实现了新的最优结果。最后，我们指出\ourmodel{}首次探索了分割与检测联合训练的潜力，期望它能作为面向开放场景下统一双任务模型的强基线方法。