Currently prevalent multimodal 3D detection methods are built upon LiDAR-based detectors that usually use dense Bird's-Eye-View (BEV) feature maps. However, the cost of such BEV feature maps is quadratic to the detection range, making it not suitable for long-range detection. Fully sparse architecture is gaining attention as they are highly efficient in long-range perception. In this paper, we study how to effectively leverage image modality in the emerging fully sparse architecture. Particularly, utilizing instance queries, our framework integrates the well-studied 2D instance segmentation into the LiDAR side, which is parallel to the 3D instance segmentation part in the fully sparse detector. This design achieves a uniform query-based fusion framework in both the 2D and 3D sides while maintaining the fully sparse characteristic. Extensive experiments showcase state-of-the-art results on the widely used nuScenes dataset and the long-range Argoverse 2 dataset. Notably, the inference speed of the proposed method under the long-range LiDAR perception setting is 2.7 $\times$ faster than that of other state-of-the-art multimodal 3D detection methods. Code will be released at \url{https://github.com/BraveGroup/FullySparseFusion}.

翻译：当前主流的模态融合3D检测方法基于激光雷达检测器，通常使用密集的鸟瞰图（BEV）特征图。然而，此类BEV特征图的计算代价随检测范围呈二次增长，不适用于长距检测。全稀疏架构因在长距感知中具有高效性而备受关注。本文研究了如何在新兴的全稀疏架构中有效利用图像模态。具体而言，我们利用实例查询机制，将成熟的2D实例分割与全稀疏检测器中的3D实例分割部分并行整合至激光雷达侧。该设计在维持全稀疏特性的同时，实现了2D与3D两侧统一的基于查询的融合框架。大量实验表明，该方法在广泛使用的nuScenes数据集以及长距Argoverse 2数据集上均取得了最先进的结果。值得注意的是，在长距激光雷达感知设置下，所提方法的推理速度相较其他现有最优模态融合3D检测方法提升2.7倍。代码将于\url{https://github.com/BraveGroup/FullySparseFusion} 公开。