We present OpenSeeD, a simple Open-vocabulary Segmentation and Detection framework that jointly learns from different segmentation and detection datasets. To bridge the gap of vocabulary and annotation granularity, we first introduce a pre-trained text encoder to encode all the visual concepts in two tasks and learn a common semantic space for them. This gives us reasonably good results compared with the counterparts trained on segmentation task only. To further reconcile them, we locate two discrepancies: $i$) task discrepancy -- segmentation requires extracting masks for both foreground objects and background stuff, while detection merely cares about the former; $ii$) data discrepancy -- box and mask annotations are with different spatial granularity, and thus not directly interchangeable. To address these issues, we propose a decoupled decoding to reduce the interference between foreground/background and a conditioned mask decoding to assist in generating masks for given boxes. To this end, we develop a simple encoder-decoder model encompassing all three techniques and train it jointly on COCO and Objects365. After pre-training, our model exhibits competitive or stronger zero-shot transferability for both segmentation and detection. Specifically, OpenSeeD beats the state-of-the-art method for open-vocabulary instance and panoptic segmentation across 5 datasets, and outperforms previous work for open-vocabulary detection on LVIS and ODinW under similar settings. When transferred to specific tasks, our model achieves new SoTA for panoptic segmentation on COCO and ADE20K, and instance segmentation on ADE20K and Cityscapes. Finally, we note that OpenSeeD is the first to explore the potential of joint training on segmentation and detection, and hope it can be received as a strong baseline for developing a single model for both tasks in open world.

翻译：我们提出OpenSeeD，一个简化的开放词汇分割与检测框架，可同时从不同的分割与检测数据集中学习。为弥合词汇与注释粒度的差异，我们首先引入预训练文本编码器对两类任务中的所有视觉概念进行编码，并为其学习统一的语义空间。与仅基于分割任务训练的同类模型相比，该方法已取得显著优异的结果。为进一步协调两类任务，我们定位了两处差异：$i$）任务差异——分割需要提取前景物体与背景区域的掩码，而检测仅关注前者；$ii$）数据差异——边界框与掩码注释具有不同的空间粒度，因此无法直接互换。为解决这些问题，我们提出解耦解码以减少前景/背景的相互干扰，以及条件掩码解码辅助生成给定边界框的掩码。最终，我们开发了一个集成上述三种技术的简洁编码器-解码器模型，并在COCO与Objects365数据集上联合训练。预训练后，该模型在分割与检测任务中展现出具有竞争力的零样本迁移能力。具体而言，OpenSeeD在5个数据集上的开放词汇实例分割与全景分割任务中超越了最先进方法，并在类似设置下于LVIS与ODinW数据集上的开放词汇检测中优于先前工作。迁移至特定任务时，该模型在COCO与ADE20K的全景分割、ADE20K与Cityscapes的实例分割任务上均达到新最优水平。最后，我们指出OpenSeeD是首个探索分割与检测联合训练潜力的工作，希望其能作为开放世界中同时处理两类任务的单模型强基线。