We address the task of view synthesis, generating novel views of a scene given a set of images as input. In many recent works such as NeRF (Mildenhall et al., 2020), the scene geometry is parameterized using neural implicit representations (i.e., MLPs). Implicit neural representations have achieved impressive visual quality but have drawbacks in computational efficiency. In this work, we propose a new approach that performs view synthesis using point clouds. It is the first point-based method that achieves better visual quality than NeRF while being 100x faster in rendering speed. Our approach builds on existing works on differentiable point-based rendering but introduces a novel technique we call "Sculpted Neural Points (SNP)", which significantly improves the robustness to errors and holes in the reconstructed point cloud. We further propose to use view-dependent point features based on spherical harmonics to capture non-Lambertian surfaces, and new designs in the point-based rendering pipeline that further boost the performance. Finally, we show that our system supports fine-grained scene editing. Code is available at https://github.com/princeton-vl/SNP.

翻译：我们解决了视图合成任务，即根据输入的一组图像生成场景的新视角。在近期许多工作（如NeRF，Mildenhall等人，2020）中，场景几何通过神经隐式表示（即MLP）进行参数化。神经隐式表示在视觉质量上取得了令人瞩目的成果，但在计算效率方面存在不足。在本工作中，我们提出了一种利用点云进行视图合成的新方法。这是首个在视觉质量上优于NeRF且渲染速度快100倍的基于点的方法。我们的方法建立在现有可微分点渲染工作的基础上，同时引入了一种名为“雕刻神经点（SNP）”的新技术，该技术显著提升了对重建点云中误差和空洞的鲁棒性。我们进一步提出基于球谐函数的视角相关点特征以捕获非朗伯表面，并对点渲染管道进行创新设计以提升性能。最后，我们展示了系统支持细粒度的场景编辑。代码已开源：https://github.com/princeton-vl/SNP。