Diminished reality (DR) refers to the removal of real objects from the environment by virtually replacing them with their background. Modern DR frameworks use inpainting to hallucinate unobserved regions. While recent deep learning-based inpainting is promising, the DR use case is complicated by the need to generate coherent structure and 3D geometry (i.e., depth), in particular for advanced applications, such as 3D scene editing. In this paper, we propose DeepDR, a first RGB-D inpainting framework fulfilling all requirements of DR: Plausible image and geometry inpainting with coherent structure, running at real-time frame rates, with minimal temporal artifacts. Our structure-aware generative network allows us to explicitly condition color and depth outputs on the scene semantics, overcoming the difficulty of reconstructing sharp and consistent boundaries in regions with complex backgrounds. Experimental results show that the proposed framework can outperform related work qualitatively and quantitatively.

翻译：缩减现实（DR）是指通过虚拟替换背景来移除真实场景中对象的技术。现代DR框架利用修复方法推断未观测区域。尽管基于深度学习的修复技术前景广阔，但DR应用场景需生成连贯结构及三维几何（即深度），尤其对于三维场景编辑等高级应用而言更具复杂性。本文提出DeepDR——首个满足DR所有要求的RGB-D修复框架：实现具有连贯结构的图像与几何可信修复，支持实时帧率运行，且时序伪影最小化。我们的结构感知生成网络能够显式地将颜色与深度输出条件化于场景语义，克服了复杂背景区域中重建清晰一致边界的难题。实验结果表明，所提框架在定性与定量分析中均优于现有相关工作。