Neural Radiance Field (NeRF) has achieved substantial progress in novel view synthesis given multi-view images. Recently, some works have attempted to train a NeRF from a single image with 3D priors. They mainly focus on a limited field of view with a few occlusions, which greatly limits their scalability to real-world 360-degree panoramic scenarios with large-size occlusions. In this paper, we present PERF, a 360-degree novel view synthesis framework that trains a panoramic neural radiance field from a single panorama. Notably, PERF allows 3D roaming in a complex scene without expensive and tedious image collection. To achieve this goal, we propose a novel collaborative RGBD inpainting method and a progressive inpainting-and-erasing method to lift up a 360-degree 2D scene to a 3D scene. Specifically, we first predict a panoramic depth map as initialization given a single panorama and reconstruct visible 3D regions with volume rendering. Then we introduce a collaborative RGBD inpainting approach into a NeRF for completing RGB images and depth maps from random views, which is derived from an RGB Stable Diffusion model and a monocular depth estimator. Finally, we introduce an inpainting-and-erasing strategy to avoid inconsistent geometry between a newly-sampled view and reference views. The two components are integrated into the learning of NeRFs in a unified optimization framework and achieve promising results. Extensive experiments on Replica and a new dataset PERF-in-the-wild demonstrate the superiority of our PERF over state-of-the-art methods. Our PERF can be widely used for real-world applications, such as panorama-to-3D, text-to-3D, and 3D scene stylization applications. Project page and code are available at https://perf-project.github.io/ and https://github.com/perf-project/PeRF.

翻译：摘要：神经辐射场（NeRF）在多视角图像的新视角合成任务中取得了显著进展。近期，部分研究尝试利用三维先验从单张图像训练NeRF，但这些方法主要聚焦于视野有限、遮挡较少的场景，严重制约了其在存在大规模遮挡的真实360度全景场景中的可扩展性。本文提出PERF——一种基于单张全景图训练全景神经辐射场的360度新视角合成框架。值得注意的是，PERF能在无需昂贵且繁琐的图像采集流程下实现复杂场景的三维漫游。为实现该目标，我们提出一种新型协作式RGBD补全方法和渐进式补全-擦除策略，将360度二维场景升维至三维场景。具体而言，首先以单张全景图为输入预测全景深度图作为初始化，通过体渲染重建可见三维区域；随后将基于RGB稳定扩散模型与单目深度估计器的协作式RGBD补全方法引入NeRF，从随机视角补全RGB图像与深度图；最后采用补全-擦除策略避免新采样视角与参考视角间的几何不一致性。上述两个组件被集成至统一的优化框架中协同学习NeRF，并取得显著效果。在Replica数据集及新构建的PERF-in-the-wild数据集上的大量实验表明，PERF相较现有最优方法具有显著优势。PERF可广泛应用于全景图转三维、文本转三维及三维场景风格化等真实场景应用。项目主页与代码分别发布于https://perf-project.github.io/ 和 https://github.com/perf-project/PeRF。