There are many recent research efforts to fine-tune a pre-trained generator with a few target images to generate images of a novel domain. Unfortunately, these methods often suffer from overfitting or under-fitting when fine-tuned with a single target image. To address this, here we present a novel single-shot GAN adaptation method through unified CLIP space manipulations. Specifically, our model employs a two-step training strategy: reference image search in the source generator using a CLIP-guided latent optimization, followed by generator fine-tuning with a novel loss function that imposes CLIP space consistency between the source and adapted generators. To further improve the adapted model to produce spatially consistent samples with respect to the source generator, we also propose contrastive regularization for patchwise relationships in the CLIP space. Experimental results show that our model generates diverse outputs with the target texture and outperforms the baseline models both qualitatively and quantitatively. Furthermore, we show that our CLIP space manipulation strategy allows more effective attribute editing.

翻译：近年来，许多研究尝试通过少量目标图像对预训练生成器进行微调，以生成新领域的图像。然而，当仅使用单张目标图像进行微调时，这些方法常面临过拟合或欠拟合问题。为解决此问题，本文提出一种通过统一CLIP空间操作实现的单样本GAN自适应方法。具体而言，我们的模型采用两步训练策略：首先利用CLIP引导的潜在空间优化在源生成器中搜索参考图像，随后通过引入保持源生成器与自适应生成器之间CLIP空间一致性的新型损失函数对生成器进行微调。为提升自适应模型生成与源生成器空间一致样本的能力，我们进一步提出基于CLIP空间块级关系的对比正则化方法。实验结果表明，我们的模型能够生成具有目标纹理的多样化输出，并在定性和定量评估中均优于基线模型。此外，我们证实这种CLIP空间操作策略可实现更高效的属性编辑。