We introduce a new task, novel view synthesis for LiDAR sensors. While traditional model-based LiDAR simulators with style-transfer neural networks can be applied to render novel views, they fall short in producing accurate and realistic LiDAR patterns, because the renderers they rely on exploit game engines, which are not differentiable. We address this by formulating, to the best of our knowledge, the first differentiable LiDAR renderer, and propose an end-to-end framework, LiDAR-NeRF, leveraging a neural radiance field (NeRF) to enable jointly learning the geometry and the attributes of 3D points. To evaluate the effectiveness of our approach, we establish an object-centric multi-view LiDAR dataset, dubbed NeRF-MVL. It contains observations of objects from 9 categories seen from 360-degree viewpoints captured with multiple LiDAR sensors. Our extensive experiments on the scene-level KITTI-360 dataset, and on our object-level NeRF-MVL show that our LiDAR- NeRF surpasses the model-based algorithms significantly.

翻译：我们提出了一项新任务：针对激光雷达传感器的新型视角合成。尽管传统的基于模型的激光雷达模拟器结合风格迁移神经网络可用于渲染新视角，但由于其依赖的渲染器利用游戏引擎而不可微分，因此在生成准确且逼真的激光雷达点云模式方面存在不足。为解决这一问题，我们首次提出了可微分的激光雷达渲染器，并构建了端到端框架LiDAR-NeRF，该框架利用神经辐射场（NeRF）实现了3D点几何与属性的联合学习。为评估方法的有效性，我们建立了一个以物体为中心的多视角激光雷达数据集NeRF-MVL，该数据集包含使用多个激光雷达传感器从360度视角捕获的9类物体观测数据。在场景级KITTI-360数据集与物体级NeRF-MVL数据集上的大量实验表明，我们的LiDAR-NeRF显著优于基于模型的算法。