Text-guided image inpainting (TGII) aims to restore missing regions based on a given text in a damaged image. Existing methods are based on a strong vision encoder and a cross-modal fusion model to integrate cross-modal features. However, these methods allocate most of the computation to visual encoding, while light computation on modeling modality interactions. Moreover, they take cross-modal fusion for depth features, which ignores a fine-grained alignment between text and image. Recently, vision-language pre-trained models (VLPM), encapsulating rich cross-modal alignment knowledge, have advanced in most multimodal tasks. In this work, we propose a novel model for TGII by improving cross-modal alignment (CMA). CMA model consists of a VLPM as a vision-language encoder, an image generator and global-local discriminators. To explore cross-modal alignment knowledge for image restoration, we introduce cross-modal alignment distillation and in-sample distribution distillation. In addition, we employ adversarial training to enhance the model to fill the missing region in complicated structures effectively. Experiments are conducted on two popular vision-language datasets. Results show that our model achieves state-of-the-art performance compared with other strong competitors.

翻译：文本引导图像修复（TGII）旨在根据受损图像中的给定文本恢复缺失区域。现有方法基于强大的视觉编码器和跨模态融合模型来整合跨模态特征。然而，这些方法将大部分计算分配给视觉编码，而对模态交互建模的计算较轻。此外，它们对深度特征进行跨模态融合，忽略了文本与图像之间的细粒度对齐。近年来，封装了丰富跨模态对齐知识的视觉-语言预训练模型（VLPM）在大多数多模态任务中取得了进展。本文提出一种通过改进跨模态对齐（CMA）的新模型用于TGII。CMA模型由作为视觉-语言编码器的VLPM、图像生成器和全局-局部判别器组成。为探索图像恢复中的跨模态对齐知识，我们引入了跨模态对齐蒸馏和样本内分布蒸馏。此外，我们采用对抗训练增强模型在复杂结构中有效填充缺失区域的能力。在两个流行的视觉-语言数据集上进行了实验。结果表明，与其他强竞争对手相比，我们的模型达到了最先进的性能。