In the last few years, deep neural networks opened the doors for big advances in novel view synthesis. Many of these approaches are based on a (coarse) proxy geometry obtained by structure from motion algorithms. Small deficiencies in this proxy can be fixed by neural rendering, but larger holes or missing parts, as they commonly appear for thin structures or for glossy regions, still lead to distracting artifacts and temporal instability. In this paper, we present a novel neural-rendering-based approach to detect and fix such deficiencies. As a proxy, we use a point cloud, which allows us to easily remove outlier geometry and to fill in missing geometry without complicated topological operations. Keys to our approach are (i) a differentiable, blending point-based renderer that can blend out redundant points, as well as (ii) the concept of Visual Error Tomography (VET), which allows us to lift 2D error maps to identify 3D-regions lacking geometry and to spawn novel points accordingly. Furthermore, (iii) by adding points as nested environment maps, our approach allows us to generate high-quality renderings of the surroundings in the same pipeline. In our results, we show that our approach can improve the quality of a point cloud obtained by structure from motion and thus increase novel view synthesis quality significantly. In contrast to point growing techniques, the approach can also fix large-scale holes and missing thin structures effectively. Rendering quality outperforms state-of-the-art methods and temporal stability is significantly improved, while rendering is possible at real-time frame rates.

翻译：摘要：近年来，深度神经网络为新颖视角合成领域的重大突破打开了大门。许多此类方法基于运动恢复结构算法获得的（粗略）代理几何体。微小的代理缺陷可通过神经渲染修复，但薄结构或高光区域常见的较大孔洞或缺失部分仍会导致视觉伪影和时序不稳定性。本文提出一种基于神经渲染的新方法来检测并修复此类缺陷。我们采用点云作为代理几何体，既可轻松移除异常几何结构，又能在不涉及复杂拓扑操作的情况下填补缺失几何结构。本方法的核心要素包括：(i) 可微分的混合点基渲染器，能消除冗余点；(ii) 视觉误差断层扫描（VET）概念，通过提升二维误差图识别缺乏几何结构的三维区域并相应生成新点。此外，(iii) 通过将点作为嵌套环境贴图添加，本方法可在同一流程中生成高质量的环境渲染。实验结果表明，本方法能显著提升运动恢复结构点云的质量，从而大幅提高新颖视角合成质量。与点生长技术不同，本方法能有效修复大尺度孔洞和缺失薄结构。渲染质量超越现有最优方法，时序稳定性显著提升，且支持实时帧率渲染。