Transformer based methods have achieved great success in image inpainting recently. However, we find that these solutions regard each pixel as a token, thus suffering from an information loss issue from two aspects: 1) They downsample the input image into much lower resolutions for efficiency consideration. 2) They quantize $256^3$ RGB values to a small number (such as 512) of quantized color values. The indices of quantized pixels are used as tokens for the inputs and prediction targets of the transformer. To mitigate these issues, we propose a new transformer based framework called "PUT". Specifically, to avoid input downsampling while maintaining computation efficiency, we design a patch-based auto-encoder P-VQVAE. The encoder converts the masked image into non-overlapped patch tokens and the decoder recovers the masked regions from the inpainted tokens while keeping the unmasked regions unchanged. To eliminate the information loss caused by input quantization, an Un-quantized Transformer is applied. It directly takes features from the P-VQVAE encoder as input without any quantization and only regards the quantized tokens as prediction targets. Furthermore, to make the inpainting process more controllable, we introduce semantic and structural conditions as extra guidance. Extensive experiments show that our method greatly outperforms existing transformer based methods on image fidelity and achieves much higher diversity and better fidelity than state-of-the-art pluralistic inpainting methods on complex large-scale datasets (e.g., ImageNet). Codes are available at https://github.com/liuqk3/PUT.

翻译：基于Transformer的方法最近在图像修复中取得了巨大成功。然而，我们发现这些方法将每个像素视为一个令牌，从而在两方面遭受信息损失问题：1）为考虑效率，它们将输入图像降采样到更低的分辨率；2）它们将$256^3$种RGB值量化成少量（如512个）量化颜色值，量化像素的索引被用作Transformer的输入令牌和预测目标。为缓解这些问题，我们提出了名为“PUT”的新型Transformer框架。具体而言，为避免输入降采样同时保持计算效率，我们设计了基于块的自动编码器P-VQVAE，其编码器将掩码图像转换为非重叠的块令牌，解码器从补全后的令牌恢复掩码区域，同时保持未掩码区域不变。为消除输入量化导致的信息损失，我们应用了非量化Transformer（Un-quantized Transformer），它直接采用P-VQVAE编码器的特征作为输入而不进行任何量化，仅将量化后的令牌作为预测目标。此外，为使修复过程更可控，我们引入了语义和结构条件作为额外引导。大量实验表明，我们的方法在图像保真度上显著优于现有基于Transformer的方法，并在复杂大规模数据集（如ImageNet）上，相比最先进的多元修复方法实现了更高的多样性和更好的保真度。代码开源在https://github.com/liuqk3/PUT。