The evolution of semantic segmentation has long been dominated by learning more discriminative image representations for classifying each pixel. Despite the prominent advancements, the priors of segmentation masks themselves, e.g., geometric and semantic constraints, are still under-explored. In this paper, we propose to ameliorate the semantic segmentation quality of existing discriminative approaches with a mask prior modeled by a recently-developed denoising diffusion generative model. Beginning with a unified architecture that adapts diffusion models for mask prior modeling, we focus this work on a specific instantiation with discrete diffusion and identify a variety of key design choices for its successful application. Our exploratory analysis revealed several important findings, including: (1) a simple integration of diffusion models into semantic segmentation is not sufficient, and a poorly-designed diffusion process might lead to degradation in segmentation performance; (2) during the training, the object to which noise is added is more important than the type of noise; (3) during the inference, the strict diffusion denoising scheme may not be essential and can be relaxed to a simpler scheme that even works better. We evaluate the proposed prior modeling with several off-the-shelf segmentors, and our experimental results on ADE20K and Cityscapes demonstrate that our approach could achieve competitively quantitative performance and more appealing visual quality.

翻译：语义分割的发展长期以来主要依赖于学习更具判别性的图像表示以对每个像素进行分类。尽管取得了显著进展，但分割掩码本身的先验知识（例如几何约束和语义约束）仍未得到充分探索。本文提出利用最近发展的去噪扩散生成模型所建模的掩码先验，改进现有判别式方法的语义分割质量。我们从适配扩散模型进行掩码先验建模的统一架构出发，重点研究基于离散扩散的特定实例，并识别出成功应用该方法的若干关键设计选择。探索性分析揭示了多项重要发现，包括：（1）将扩散模型简单集成到语义分割中并不足够，设计不当的扩散过程可能导致分割性能下降；（2）在训练过程中，添加噪声的对象比噪声类型更为关键；（3）在推理过程中，严格的扩散去噪方案并非必要，可简化为一种更简单的方案，且效果更优。本文使用多种现成分割器对所提出的先验建模方法进行评估，在ADE20K和Cityscapes上的实验结果表明，我们的方法能够实现具有竞争力的量化性能及更优的视觉质量。