Diffusion models have demonstrated excellent performance in image generation. Although various few-shot semantic segmentation (FSS) models with different network structures have been proposed, performance improvement has reached a bottleneck. This paper presents the first work to leverage the diffusion model for FSS task, called DifFSS. DifFSS, a novel FSS paradigm, can further improve the performance of the state-of-the-art FSS models by a large margin without modifying their network structure. Specifically, we utilize the powerful generation ability of diffusion models to generate diverse auxiliary support images by using the semantic mask, scribble or soft HED boundary of the support image as control conditions. This generation process simulates the variety within the class of the query image, such as color, texture variation, lighting, $etc$. As a result, FSS models can refer to more diverse support images, yielding more robust representations, thereby achieving a consistent improvement in segmentation performance. Extensive experiments on three publicly available datasets based on existing advanced FSS models demonstrate the effectiveness of the diffusion model for FSS task. Furthermore, we explore in detail the impact of different input settings of the diffusion model on segmentation performance. Hopefully, this completely new paradigm will bring inspiration to the study of FSS task integrated with AI-generated content.

翻译：扩散模型在图像生成领域展现出卓越性能。尽管现有不同网络结构的小样本语义分割（FSS）模型已层出不穷，但其性能提升已进入瓶颈期。本文首次提出将扩散模型应用于FSS任务，命名为DifFSS。作为一种新型FSS范式，DifFSS无需修改现有先进FSS模型的网络结构，即可大幅提升其分割性能。具体而言，我们利用扩散模型的强大生成能力，以支撑图像的语义掩码、涂鸦或软HED边界作为控制条件，生成多样化的辅助支撑图像。此生成过程模拟了查询图像类内多样性，例如颜色、纹理变化及光照条件$等方面$。因此，FSS模型可参考更丰富的支撑图像，获得更具鲁棒性的表征，从而持续提升分割性能。基于现有先进FSS模型在三个公开数据集上的大量实验，验证了扩散模型在FSS任务中的有效性。此外，我们深入探究了扩散模型不同输入设置对分割性能的影响。该全新范式有望为融合人工智能生成内容的FSS任务研究带来启发。