3D reconstruction from a single-view is challenging because of the ambiguity from monocular cues and lack of information about occluded regions. Neural radiance fields (NeRF), while popular for view synthesis and 3D reconstruction, are typically reliant on multi-view images. Existing methods for single-view 3D reconstruction with NeRF rely on either data priors to hallucinate views of occluded regions, which may not be physically accurate, or shadows observed by RGB cameras, which are difficult to detect in ambient light and low albedo backgrounds. We propose using time-of-flight data captured by a single-photon avalanche diode to overcome these limitations. Our method models two-bounce optical paths with NeRF, using lidar transient data for supervision. By leveraging the advantages of both NeRF and two-bounce light measured by lidar, we demonstrate that we can reconstruct visible and occluded geometry without data priors or reliance on controlled ambient lighting or scene albedo. In addition, we demonstrate improved generalization under practical constraints on sensor spatial- and temporal-resolution. We believe our method is a promising direction as single-photon lidars become ubiquitous on consumer devices, such as phones, tablets, and headsets.

翻译：单视角三维重建因单目线索的模糊性以及遮挡区域信息缺失而极具挑战性。神经辐射场（NeRF）虽在视图合成与三维重建领域广受欢迎，但通常依赖多视角图像。现有基于NeRF的单视角三维重建方法，要么依赖数据先验来推测遮挡区域的视图（可能缺乏物理精确性），要么依赖RGB相机观测到的阴影（在环境光与低反照率背景下难以检测）。我们提出利用单光子雪崩二极管获取的飞行时间数据来克服这些局限。本方法通过NeRF建模双次弹射光路，并以激光雷达瞬态数据作为监督信号。通过融合NeRF与激光雷达双次弹射光测量的优势，我们证明无需数据先验、无需依赖受控环境光照或场景反照率，即可重建可见与遮挡几何结构。此外，我们展示了该方法在传感器空间与时间分辨率的实际约束下仍具有更优的泛化能力。随着单光子激光雷达在手机、平板、头显等消费设备中普及，我们认为本方法具有广阔前景。