Diffusion models have demonstrated excellent performance in image generation. Although various few-shot semantic segmentation (FSS) models with different network structures have been proposed, performance improvement has reached a bottleneck. This paper presents the first work to leverage the diffusion model for FSS task, called DifFSS. DifFSS, a novel FSS paradigm, can further improve the performance of the state-of-the-art FSS models by a large margin without modifying their network structure. Specifically, we utilize the powerful generation ability of diffusion models to generate diverse auxiliary support images by using the semantic mask, scribble or soft HED boundary of the support image as control conditions. This generation process simulates the variety within the class of the query image, such as color, texture variation, lighting, $etc$. As a result, FSS models can refer to more diverse support images, yielding more robust representations, thereby achieving a consistent improvement in segmentation performance. Extensive experiments on three publicly available datasets based on existing advanced FSS models demonstrate the effectiveness of the diffusion model for FSS task. Furthermore, we explore in detail the impact of different input settings of the diffusion model on segmentation performance. Hopefully, this completely new paradigm will bring inspiration to the study of FSS task integrated with AI-generated content.

翻译：扩散模型在图像生成领域展现出卓越性能。尽管已有多种不同网络结构的少样本语义分割（FSS）模型被提出，但其性能提升已进入瓶颈期。本文首次提出将扩散模型应用于FSS任务的工作，称为DifFSS。DifFSS作为一种新型FSS范式，能够在无需修改现有最优FSS模型网络结构的前提下，大幅提升其性能。具体而言，我们利用扩散模型强大的生成能力，以支撑图像的语义掩码、涂鸦标注或软HED边界作为控制条件，生成多样化的辅助支撑图像。该生成过程模拟了查询图像类内多样性（如颜色、纹理变化、光照等）。由此，FSS模型可参考更多样化的支撑图像，获得更具鲁棒性的表征，从而持续提升分割性能。基于现有先进FSS模型在三个公开数据集上的大量实验表明，扩散模型对FSS任务具有显著有效性。此外，我们深入探究了扩散模型不同输入设置对分割性能的影响。这一全新范式有望为融合人工智能生成内容的FSS研究带来启发。