Neural Radiance Fields (NeRF) have shown impressive novel view synthesis results; nonetheless, even thorough recordings yield imperfections in reconstructions, for instance due to poorly observed areas or minor lighting changes. Our goal is to mitigate these imperfections from various sources with a joint solution: we take advantage of the ability of generative adversarial networks (GANs) to produce realistic images and use them to enhance realism in 3D scene reconstruction with NeRFs. To this end, we learn the patch distribution of a scene using an adversarial discriminator, which provides feedback to the radiance field reconstruction, thus improving realism in a 3D-consistent fashion. Thereby, rendering artifacts are repaired directly in the underlying 3D representation by imposing multi-view path rendering constraints. In addition, we condition a generator with multi-resolution NeRF renderings which is adversarially trained to further improve rendering quality. We demonstrate that our approach significantly improves rendering quality, e.g., nearly halving LPIPS scores compared to Nerfacto while at the same time improving PSNR by 1.4dB on the advanced indoor scenes of Tanks and Temples.

翻译：神经辐射场（NeRF）在新视角合成方面取得了令人瞩目的成果；然而，即使是全面的记录也难以避免重建中的瑕疵，例如由于观测不佳的区域或微小的光照变化所致。我们的目标是借助一个联合解决方案来缓解来自不同来源的这些瑕疵：利用生成对抗网络（GAN）生成逼真图像的能力，并将其用于增强基于NeRF的三维场景重建的真实感。为此，我们使用对抗性判别器学习场景的块分布，该判别器为辐射场重建提供反馈，从而以三维一致的方式提升真实感。通过施加多视角路径渲染约束，渲染伪影可直接在底层三维表示中得到修复。此外，我们以多分辨率NeRF渲染结果作为条件输入，训练一个生成器以对抗性方式进一步提升渲染质量。实验表明，我们的方法显著提高了渲染质量：在Tanks and Temples的高级室内场景中，与Nerfacto相比，LPIPS分数几乎减半，同时PSNR提升了1.4dB。