Training perception systems for self-driving cars requires substantial annotations. However, manual labeling in 2D images is highly labor-intensive. While existing datasets provide rich annotations for pre-recorded sequences, they fall short in labeling rarely encountered viewpoints, potentially hampering the generalization ability for perception models. In this paper, we present PanopticNeRF-360, a novel approach that combines coarse 3D annotations with noisy 2D semantic cues to generate consistent panoptic labels and high-quality images from any viewpoint. Our key insight lies in exploiting the complementarity of 3D and 2D priors to mutually enhance geometry and semantics. Specifically, we propose to leverage noisy semantic and instance labels in both 3D and 2D spaces to guide geometry optimization. Simultaneously, the improved geometry assists in filtering noise present in the 3D and 2D annotations by merging them in 3D space via a learned semantic field. To further enhance appearance, we combine MLP and hash grids to yield hybrid scene features, striking a balance between high-frequency appearance and predominantly contiguous semantics. Our experiments demonstrate PanopticNeRF-360's state-of-the-art performance over existing label transfer methods on the challenging urban scenes of the KITTI-360 dataset. Moreover, PanopticNeRF-360 enables omnidirectional rendering of high-fidelity, multi-view and spatiotemporally consistent appearance, semantic and instance labels. We make our code and data available at https://github.com/fuxiao0719/PanopticNeRF

翻译：自动驾驶感知系统的训练需要大量标注数据。然而，对二维图像进行人工标注极其耗费人力。尽管现有数据集为预先录制的序列提供了丰富的标注，但在处理罕见视角的标注时仍显不足，这可能制约感知模型的泛化能力。本文提出PanopticNeRF-360这一新颖方法，通过结合粗糙的三维标注与含噪声的二维语义线索，从任意视角生成一致性全景标签与高质量图像。我们的核心洞见在于利用三维与二维先验的互补性，以协同优化几何与语义。具体而言，我们提出利用三维和二维空间中的噪声语义与实例标签引导几何优化，同时改进后的几何通过将三维与二维标注在三维空间中融合（借助学习到的语义场）来辅助过滤其中的噪声。为增强外观表现，我们结合MLP与哈希网格生成混合场景特征，在高频外观与连续性占主导的语义之间取得平衡。实验表明，PanopticNeRF-360在KITTI-360数据集的挑战性城市场景中，相较于现有标签迁移方法取得了最优性能。此外，PanopticNeRF-360实现了高保真、多视角且时空一致的外观、语义与实例标签的全向渲染。我们的代码与数据已开源至https://github.com/fuxiao0719/PanopticNeRF。