Multi-view camera-based 3D detection is a challenging problem in computer vision. Recent works leverage a pretrained LiDAR detection model to transfer knowledge to a camera-based student network. However, we argue that there is a major domain gap between the LiDAR BEV features and the camera-based BEV features, as they have different characteristics and are derived from different sources. In this paper, we propose Geometry Enhanced Masked Image Modeling (GeoMIM) to transfer the knowledge of the LiDAR model in a pretrain-finetune paradigm for improving the multi-view camera-based 3D detection. GeoMIM is a multi-camera vision transformer with Cross-View Attention (CVA) blocks that uses LiDAR BEV features encoded by the pretrained BEV model as learning targets. During pretraining, GeoMIM's decoder has a semantic branch completing dense perspective-view features and the other geometry branch reconstructing dense perspective-view depth maps. The depth branch is designed to be camera-aware by inputting the camera's parameters for better transfer capability. Extensive results demonstrate that GeoMIM outperforms existing methods on nuScenes benchmark, achieving state-of-the-art performance for camera-based 3D object detection and 3D segmentation.

翻译：多视图基于相机的三维检测是计算机视觉中的一项挑战性问题。近期研究利用预训练的激光雷达检测模型，将知识迁移至基于相机的学生网络。然而，我们认为激光雷达BEV特征与基于相机的BEV特征之间存在显著的领域差距，因为它们具有不同的特性且源自不同来源。本文提出几何增强掩码图像建模（GeoMIM），通过预训练-微调范式迁移激光雷达模型的知识，以提升多视图基于相机的三维检测性能。GeoMIM是一个多相机视觉Transformer，包含跨视图注意力（CVA）模块，并以预训练BEV模型编码的激光雷达BEV特征作为学习目标。在预训练过程中，GeoMIM的解码器包含一个语义分支以完成密集透视图特征，以及一个几何分支以重建密集透视图深度图。深度分支通过输入相机参数设计为相机感知，以实现更好的迁移能力。大量实验结果表明，GeoMIM在nuScenes基准上超越现有方法，在基于相机的三维目标检测和三维分割任务中均达到最先进性能。