Observing the close relationship among panoptic, semantic and instance segmentation tasks, we propose to train a universal multi-dataset multi-task segmentation model: DaTaSeg.We use a shared representation (mask proposals with class predictions) for all tasks. To tackle task discrepancy, we adopt different merge operations and post-processing for different tasks. We also leverage weak-supervision, allowing our segmentation model to benefit from cheaper bounding box annotations. To share knowledge across datasets, we use text embeddings from the same semantic embedding space as classifiers and share all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO panoptic, and Objects365 detection datasets. DaTaSeg improves performance on all datasets, especially small-scale datasets, achieving 54.0 mIoU on ADE semantic and 53.5 PQ on COCO panoptic. DaTaSeg also enables weakly-supervised knowledge transfer on ADE panoptic and Objects365 instance segmentation. Experiments show DaTaSeg scales with the number of training datasets and enables open-vocabulary segmentation through direct transfer. In addition, we annotate an Objects365 instance segmentation set of 1,000 images and will release it as a public benchmark.

翻译：摘要：观察到全景分割、语义分割和实例分割任务之间的紧密关系，我们提出训练一个通用的多数据集多任务分割模型：DaTaSeg。该模型对所有任务采用共享表示（带类别预测的掩码提议）。为解决任务差异，我们针对不同任务采用不同的合并操作和后处理方法。同时利用弱监督机制，使分割模型能够从更廉价的边界框标注中获益。为跨数据集共享知识，我们采用相同语义嵌入空间的文本嵌入作为分类器，并在所有数据集间共享全部网络参数。我们在ADE语义分割、COCO全景分割和Objects365检测数据集上训练DaTaSeg。实验表明，DaTaSeg在所有数据集上均提升性能，尤其在小规模数据集上表现显著，在ADE语义分割上达到54.0 mIoU，在COCO全景分割上达到53.5 PQ。DaTaSeg还支持在ADE全景分割和Objects365实例分割任务上的弱监督知识迁移。实验证明，DaTaSeg的性能随训练数据集数量扩展，并通过直接迁移实现开放词汇分割。此外，我们标注了包含1000张图像的Objects365实例分割数据集，将作为公开基准发布。