This paper introduces an approach, named DFormer, for universal image segmentation. The proposed DFormer views universal image segmentation task as a denoising process using a diffusion model. DFormer first adds various levels of Gaussian noise to ground-truth masks, and then learns a model to predict denoising masks from corrupted masks. Specifically, we take deep pixel-level features along with the noisy masks as inputs to generate mask features and attention masks, employing diffusion-based decoder to perform mask prediction gradually. At inference, our DFormer directly predicts the masks and corresponding categories from a set of randomly-generated masks. Extensive experiments reveal the merits of our proposed contributions on different image segmentation tasks: panoptic segmentation, instance segmentation, and semantic segmentation. Our DFormer outperforms the recent diffusion-based panoptic segmentation method Pix2Seq-D with a gain of 3.6% on MS COCO val2017 set. Further, DFormer achieves promising semantic segmentation performance outperforming the recent diffusion-based method by 2.2% on ADE20K val set. Our source code and models will be publicly on https://github.com/cp3wan/DFormer

翻译：本文提出了一种名为DFormer的通用图像分割方法。所提出的DFormer将通用图像分割任务视为利用扩散模型的去噪过程。DFormer首先向真实标注掩码添加不同级别的高斯噪声，然后学习一个模型，从受噪声污染的掩码中预测去噪掩码。具体而言，我们以深层像素级特征和噪声掩码为输入，生成掩码特征和注意力掩码，并采用基于扩散的解码器逐步进行掩码预测。在推理阶段，我们的DFormer直接从一组随机生成的掩码中预测掩码及对应类别。大量实验揭示了我们所提出贡献在不同图像分割任务（全景分割、实例分割和语义分割）中的优势。我们的DFormer在MS COCO val2017数据集上以3.6%的提升超越了近期基于扩散的全景分割方法Pix2Seq-D。此外，DFormer在ADE20K验证集上取得了有前景的语义分割性能，比近期基于扩散的方法高出2.2%。我们的源代码和模型将公开在https://github.com/cp3wan/DFormer。