This work proposes a unified self-supervised pre-training framework for transferable multi-modal perception representation learning via masked multi-modal reconstruction in Neural Radiance Field (NeRF), namely NeRF-Supervised Masked AutoEncoder (NS-MAE). Specifically, conditioned on certain view directions and locations, multi-modal embeddings extracted from corrupted multi-modal input signals, i.e., Lidar point clouds and images, are rendered into projected multi-modal feature maps via neural rendering. Then, original multi-modal signals serve as reconstruction targets for the rendered multi-modal feature maps to enable self-supervised representation learning. Extensive experiments show that the representation learned via NS-MAE shows promising transferability for diverse multi-modal and single-modal (camera-only and Lidar-only) perception models on diverse 3D perception downstream tasks (3D object detection and BEV map segmentation) with diverse amounts of fine-tuning labeled data. Moreover, we empirically find that NS-MAE enjoys the synergy of both the mechanism of masked autoencoder and neural radiance field. Our code shall be released upon acceptance.

翻译：本工作提出了一种统一的自我监督预训练框架，用于通过神经辐射场中的掩码多模态重建实现可迁移的多模态感知表示学习，即NeRF监督掩码自编码器。具体而言，在特定视角方向和位置的条件下，从受损的多模态输入信号（即激光雷达点云和图像）中提取的多模态嵌入，通过神经渲染被渲染成投影的多模态特征图。随后，原始多模态信号作为渲染后多模态特征图的重建目标，以实现自我监督的表示学习。大量实验表明，通过NS-MAE学习到的表示在多种3D感知下游任务（3D目标检测和鸟瞰图地图分割）中，对多种多模态和单模态（仅相机和仅激光雷达）感知模型展现出良好的可迁移性，且所需微调标注数据量多样化。此外，我们实验发现NS-MAE兼具掩码自编码器与神经辐射场机制的协同优势。我们的代码将在论文被接收后公布。