Inpainting involves filling in missing pixels or areas in an image, a crucial technique employed in Mixed Reality environments for various applications, particularly in Diminished Reality (DR) where content is removed from a user's visual environment. Existing methods rely on digital replacement techniques which necessitate multiple cameras and incur high costs. AR devices and smartphones use ToF depth sensors to capture scene depth maps aligned with RGB images. Despite speed and affordability, ToF cameras create imperfect depth maps with missing pixels. To address the above challenges, we propose Hierarchical Inpainting GAN (HI-GAN), a novel approach comprising three GANs in a hierarchical fashion for RGBD inpainting. EdgeGAN and LabelGAN inpaint masked edge and segmentation label images respectively, while CombinedRGBD-GAN combines their latent representation outputs and performs RGB and Depth inpainting. Edge images and particularly segmentation label images as auxiliary inputs significantly enhance inpainting performance by complementary context and hierarchical optimization. We believe we make the first attempt to incorporate label images into inpainting process.Unlike previous approaches requiring multiple sequential models and separate outputs, our work operates in an end-to-end manner, training all three models simultaneously and hierarchically. Specifically, EdgeGAN and LabelGAN are first optimized separately and further optimized inside CombinedRGBD-GAN to enhance inpainting quality. Experiments demonstrate that HI-GAN works seamlessly and achieves overall superior performance compared with existing approaches.

翻译：摘要：图像修复涉及填补图像中的缺失像素或区域，这是混合现实环境中用于各种应用的关键技术，尤其在增强现实（Diminished Reality, DR）中，需从用户视觉环境中移除内容。现有方法依赖数字替换技术，需多摄像头支持且成本高昂。AR设备与智能手机采用飞行时间（ToF）深度传感器捕获与RGB图像对齐的场景深度图。尽管ToF摄像头速度快且价格低廉，但其生成的深度图存在像素缺失缺陷。为解决上述挑战，我们提出分层修复生成对抗网络（HI-GAN），一种由三个GAN以分层结构组成的RGBD修复新方法。EdgeGAN与LabelGAN分别修复掩蔽边缘图像与分割标签图像，而CombinedRGBD-GAN整合其潜表征输出并执行RGB与深度联合修复。边缘图像尤其是分割标签图像作为辅助输入，通过互补上下文与分层优化显著提升修复性能。我们首次将标签图像引入修复流程。与以往需多个顺序模型及独立输出的方法不同，本工作采用端到端方式，同步分层训练三个模型。具体而言，EdgeGAN与LabelGAN先独立优化，再在CombinedRGBD-GAN内部进一步优化以增强修复质量。实验表明，HI-GAN无缝运作，性能全面优于现有方法。