We present \ourmodel{}, a simple Open-vocabulary Segmentation and Detection framework that jointly learns from different segmentation and detection datasets. To bridge the gap of vocabulary and annotation granularity, we first introduce a pre-trained text encoder to encode all the visual concepts in two tasks and learn a common semantic space for them. This gives us reasonably good results compared with the counterparts trained on segmentation task only. To further reconcile them, we locate two discrepancies: $i$) task discrepancy -- segmentation requires extracting masks for both foreground objects and background stuff, while detection merely cares about the former; $ii$) data discrepancy -- box and mask annotations are with different spatial granularity, and thus not directly interchangeable. To address these issues, we propose a decoupled decoding to reduce the interference between foreground/background and a conditioned mask decoding to assist in generating masks for given boxes. To this end, we develop a simple encoder-decoder model encompassing all three techniques and train it jointly on COCO and Objects365. After pre-training, our model exhibits competitive or stronger zero-shot transferability for both segmentation and detection. Specifically, \ourmodel{} beats the state-of-the-art method for open-vocabulary instance and panoptic segmentation across 5 datasets, and outperforms previous work for open-vocabulary detection on LVIS and ODinW under similar settings. When transferred to specific tasks, our model achieves new SoTA for panoptic segmentation on COCO and ADE20K, and instance segmentation on ADE20K and Cityscapes. Finally, we note that \ourmodel{} is the first to explore the potential of joint training on segmentation and detection, and hope it can be received as a strong baseline for developing a single model for both tasks in open world.

翻译：我们提出\ourmodel{}，这是一个简单的开放词汇分割与检测框架，能够联合学习来自不同分割与检测数据集的知识。为弥合词汇与标注粒度的差异，我们首先引入预训练文本编码器，对两类任务中的所有视觉概念进行编码，并为它们学习统一的语义空间。相较于仅使用分割任务训练的同类方法，该方法取得了相当理想的结果。为进一步协调两类任务，我们定位到两个差异点：$i$)任务差异——分割需要提取前景物体与背景物质的掩码，而检测仅关注前者；$ii$)数据差异——框标注与掩码标注具有不同的空间粒度，因此无法直接互换。为解决这些问题，我们提出解耦解码机制以减少前景/背景之间的干扰，以及条件掩码解码机制来辅助为给定边界框生成掩码。基于此，我们开发了一个融合上述三种技术的简单编码器-解码器模型，并在COCO与Objects365数据集上联合训练该模型。预训练后，我们的模型在分割与检测任务上均展现出具有竞争力或更强的零样本迁移能力。具体而言，\ourmodel{}在5个数据集上的开放词汇实例分割与全景分割任务中超越了现有最优方法，并在相似设定下，于LVIS与ODinW数据集上超越此前开放词汇检测工作。迁移至特定任务后，我们的模型在COCO与ADE20K数据集上的全景分割，以及ADE20K与Cityscapes数据集上的实例分割中达到了新的最优水平。最后，我们指出\ourmodel{}是首个探索分割与检测联合训练潜在价值的工作，期待其能成为开放世界中统一完成这两类任务的强基线模型。