Text-guided image generation aimed to generate desired images conditioned on given texts, while text-guided image manipulation refers to semantically edit parts of a given image based on specified texts. For these two similar tasks, the key point is to ensure image fidelity as well as semantic consistency. Many previous approaches require complex multi-stage generation and adversarial training, while struggling to provide a unified framework for both tasks. In this work, we propose TextCLIP, a unified framework for text-guided image generation and manipulation without adversarial training. The proposed method accepts input from images or random noise corresponding to these two different tasks, and under the condition of the specific texts, a carefully designed mapping network that exploits the powerful generative capabilities of StyleGAN and the text image representation capabilities of Contrastive Language-Image Pre-training (CLIP) generates images of up to $1024\times1024$ resolution that can currently be generated. Extensive experiments on the Multi-modal CelebA-HQ dataset have demonstrated that our proposed method outperforms existing state-of-the-art methods, both on text-guided generation tasks and manipulation tasks.

翻译：文本来引导图像生成旨在根据给定文本生成目标图像，而文本来引导图像编辑则指基于指定文本对给定图像的部分区域进行语义修改。针对这两类相似任务，核心在于确保图像保真度与语义一致性。以往诸多方法需要复杂的多阶段生成与对抗训练，且难以提供统一框架处理两类任务。本文提出TextCLIP——一种无需对抗训练的统一文本引导图像生成与编辑框架。该方法针对两类任务分别接受图像或随机噪声作为输入，在特定文本条件下，通过精心设计的映射网络，利用StyleGAN强大的生成能力与对比语言-图像预训练(CLIP)的文本图像表征能力，可生成当前最高达$1024\times1024$分辨率的高质量图像。在Multi-modal CelebA-HQ数据集上的大量实验表明，本方法在文本引导生成任务与编辑任务上均优于现有最先进方法。