Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks compared with Generative Adversarial Nets (GANs). Recent work on semantic image synthesis mainly follows the de facto GAN-based approaches, which may lead to unsatisfactory quality or diversity of generated images. In this paper, we propose a novel framework based on DDPM for semantic image synthesis. Unlike previous conditional diffusion model directly feeds the semantic layout and noisy image as input to a U-Net structure, which may not fully leverage the information in the input semantic mask, our framework processes semantic layout and noisy image differently. It feeds noisy image to the encoder of the U-Net structure while the semantic layout to the decoder by multi-layer spatially-adaptive normalization operators. To further improve the generation quality and semantic interpretability in semantic image synthesis, we introduce the classifier-free guidance sampling strategy, which acknowledge the scores of an unconditional model for sampling process. Extensive experiments on four benchmark datasets demonstrate the effectiveness of our proposed method, achieving state-of-the-art performance in terms of fidelity (FID) and diversity (LPIPS). Our code and pretrained models are available at https://github.com/WeilunWang/semantic-diffusion-model.

翻译：与生成对抗网络（GANs）相比，去噪扩散概率模型（DDPMs）已在各类图像生成任务中取得显著成功。当前语义图像合成的研究主要遵循基于GAN的主流方法，这可能导致生成图像的质量或多样性不尽如人意。本文提出了一种基于DDPM的语义图像合成新框架。以往的条件扩散模型通常将语义布局与噪声图像直接输入U-Net结构，可能无法充分利用输入语义掩码的信息；而本框架对语义布局与噪声图像进行差异化处理：将噪声图像输入U-Net编码器，同时通过多层空间自适应归一化算子将语义布局注入解码器。为进一步提升语义图像合成的生成质量与语义可解释性，我们引入了无分类器引导采样策略，该策略在采样过程中融合了无条件模型的评分。在四个基准数据集上的大量实验证明了所提方法的有效性，在保真度（FID）与多样性（LPIPS）指标上均达到了最先进的性能。我们的代码与预训练模型已发布于https://github.com/WeilunWang/semantic-diffusion-model。