We propose a novel method that renders point clouds as if they are surfaces. The proposed method is differentiable and requires no scene-specific optimization. This unique capability enables, out-of-the-box, surface normal estimation, rendering room-scale point clouds, inverse rendering, and ray tracing with global illumination. Unlike existing work that focuses on converting point clouds to other representations--e.g., surfaces or implicit functions--our key idea is to directly infer the intersection of a light ray with the underlying surface represented by the given point cloud. Specifically, we train a set transformer that, given a small number of local neighbor points along a light ray, provides the intersection point, the surface normal, and the material blending weights, which are used to render the outcome of this light ray. Localizing the problem into small neighborhoods enables us to train a model with only 48 meshes and apply it to unseen point clouds. Our model achieves higher estimation accuracy than state-of-the-art surface reconstruction and point-cloud rendering methods on three test sets. When applied to room-scale point clouds, without any scene-specific optimization, the model achieves competitive quality with the state-of-the-art novel-view rendering methods. Moreover, we demonstrate ability to render and manipulate Lidar-scanned point clouds such as lighting control and object insertion.

翻译：我们提出一种新颖方法，可将点云渲染为表面。该方法具有可微性，且无需针对特定场景进行优化。这一独特能力可直接实现表面法向估计、室内尺度点云渲染、逆渲染以及全局光照光线追踪。与现有专注于将点云转换为其他表示（如表面或隐式函数）的工作不同，本文的核心思想是直接推断光线与给定点云所代表底层表面的相交。具体而言，我们训练一个集合变换器（set transformer），该模型基于沿光线方向的少量局部邻域点，输出相交点、表面法向和材质混合权重，并用于渲染该光线的最终结果。通过将问题局部化至小邻域，我们仅需48个网格即可训练模型，并直接应用于未见过的点云。在三个测试集上，我们的模型相比现有最优的表面重建和点云渲染方法实现了更高的估计精度。当应用于室内尺度点云时，无需场景级优化，该模型即能达到与前沿新视角渲染方法竞争的质量。此外，我们展示了渲染和操作激光雷达扫描点云的能力，例如光照控制与物体插入。