Image inpainting involves filling missing areas of a corrupted image. Despite impressive results have been achieved recently, restoring images with both vivid textures and reasonable structures remains a significant challenge. Previous methods have primarily addressed regular textures while disregarding holistic structures due to the limited receptive fields of Convolutional Neural Networks (CNNs). To this end, we study learning a Zero-initialized residual addition based Incremental Transformer on Structural priors (ZITS++), an improved model upon our conference work, ZITS~\cite{dong2022incremental}. Specifically, given one corrupt image, we present the Transformer Structure Restorer (TSR) module to restore holistic structural priors at low image resolution, which are further upsampled by Simple Structure Upsampler (SSU) module to higher image resolution. To recover image texture details, we use the Fourier CNN Texture Restoration (FTR) module, which is strengthened by Fourier and large-kernel attention convolutions. Furthermore, to enhance the FTR, the upsampled structural priors from TSR are further processed by Structure Feature Encoder (SFE) and optimized with the Zero-initialized Residual Addition (ZeroRA) incrementally. Besides, a new masking positional encoding is proposed to encode the large irregular masks. Compared with ZITS, ZITS++ improves the FTR's stability and inpainting ability with several techniques. More importantly, we comprehensively explore the effects of various image priors for inpainting and investigate how to utilize them to address high-resolution image inpainting with extensive experiments. This investigation is orthogonal to most inpainting approaches and can thus significantly benefit the community. Codes and models will be released in https://github.com/DQiaole/ZITS_inpainting.

翻译：图像修复涉及填充受损图像的缺失区域。尽管近期已取得显著成果，但如何同时恢复具有生动纹理与合理结构的图像仍是一项重大挑战。由于卷积神经网络（CNN）感受野的局限性，现有方法主要处理规则纹理而忽视了整体结构。为此，我们研究基于结构先验的零初始化残差相加增量式Transformer（ZITS++），这是在会议论文ZITS~\cite{dong2022incremental}基础上的改进模型。具体而言，针对一幅受损图像，我们提出Transformer结构恢复器（TSR）模块，在低图像分辨率下恢复整体结构先验，并通过简单结构上采样器（SSU）模块将其上采样至更高分辨率。为恢复图像纹理细节，我们采用傅里叶CNN纹理恢复（FTR）模块，该模块通过傅里叶和大核注意力卷积得到增强。此外，为增强FTR，TSR上采样得到的结构先验进一步经结构特征编码器（SFE）处理，并通过零初始化残差相加（ZeroRA）增量式优化。同时，提出一种新的掩码位置编码以编码大型不规则掩码。与ZITS相比，ZITS++通过多项技术提升了FTR的稳定性与修复能力。更重要的是，我们全面探索了各种图像先验对修复的影响，并通过大量实验研究如何利用这些先验解决高分辨率图像修复问题。该研究与多数修复方法正交，因此可为学界带来显著效益。代码与模型将在https://github.com/DQiaole/ZITS_inpainting发布。