Fusing LiDAR and camera information is essential for achieving accurate and reliable 3D object detection in autonomous driving systems. This is challenging due to the difficulty of combining multi-granularity geometric and semantic features from two drastically different modalities. Recent approaches aim at exploring the semantic densities of camera features through lifting points in 2D camera images (referred to as seeds) into 3D space, and then incorporate 2D semantics via cross-modal interaction or fusion techniques. However, depth information is under-investigated in these approaches when lifting points into 3D space, thus 2D semantics can not be reliably fused with 3D points. Moreover, their multi-modal fusion strategy, which is implemented as concatenation or attention, either can not effectively fuse 2D and 3D information or is unable to perform fine-grained interactions in the voxel space. To this end, we propose a novel framework with better utilization of the depth information and fine-grained cross-modal interaction between LiDAR and camera, which consists of two important components. First, a Multi-Depth Unprojection (MDU) method with depth-aware designs is used to enhance the depth quality of the lifted points at each interaction level. Second, a Gated Modality-Aware Convolution (GMA-Conv) block is applied to modulate voxels involved with the camera modality in a fine-grained manner and then aggregate multi-modal features into a unified space. Together they provide the detection head with more comprehensive features from LiDAR and camera. On the nuScenes test benchmark, our proposed method, abbreviated as MSMDFusion, achieves state-of-the-art 3D object detection results with 71.5% mAP and 74.0% NDS, and strong tracking results with 74.0% AMOTA without using test-time-augmentation and ensemble techniques. The code is available at https://github.com/SxJyJay/MSMDFusion.

翻译：融合激光雷达与摄像头信息对于实现自动驾驶系统中准确可靠的三维目标检测至关重要。然而，由于需要将两种截然不同模态中的多粒度几何与语义特征相结合，这一任务极具挑战性。现有方法尝试通过将二维摄像头图像中的点（称为种子）提升至三维空间以利用摄像头特征的语义密度，进而通过跨模态交互或融合技术整合二维语义信息。但这些方法在将点提升至三维空间时对深度信息的探索不足，导致二维语义无法与三维点实现可靠融合。此外，其以拼接或注意力机制实现的多模态融合策略，要么无法有效融合二维与三维信息，要么无法在体素空间中进行细粒度交互。为此，我们提出一种能更充分利用深度信息并实现激光雷达与摄像头间细粒度跨模态交互的新型框架，该框架包含两个关键组件：首先，采用具有深度感知设计的“多深度反投影”（MDU）方法，在每一交互层级上提升所提升点的深度质量；其次，引入“门控模态感知卷积”（GMA-Conv）模块，以细粒度方式调制包含摄像头模态的体素，并将多模态特征聚合至统一空间。两者共同为检测头提供更全面的激光雷达与摄像头特征。在nuScenes测试基准上，我们提出的方法（简称MSMDFusion）无需测试时增强与集成技术，即实现了71.5% mAP与74.0% NDS的当前最优三维目标检测结果，以及74.0% AMOTA的强跟踪结果。代码已开源至https://github.com/SxJyJay/MSMDFusion。