Robot localization using a previously built map is essential for a variety of tasks including highly accurate navigation and mobile manipulation. A popular approach to robot localization is based on image-to-point cloud registration, which combines illumination-invariant LiDAR-based mapping with economical image-based localization. However, the recent works for image-to-point cloud registration either divide the registration into separate modules or project the point cloud to the depth image to register the RGB and depth images. In this paper, we present I2PNet, a novel end-to-end 2D-3D registration network. I2PNet directly registers the raw 3D point cloud with the 2D RGB image using differential modules with a unique target. The 2D-3D cost volume module for differential 2D-3D association is proposed to bridge feature extraction and pose regression. 2D-3D cost volume module implicitly constructs the soft point-to-pixel correspondence on the intrinsic-independent normalized plane of the pinhole camera model. Moreover, we introduce an outlier mask prediction module to filter the outliers in the 2D-3D association before pose regression. Furthermore, we propose the coarse-to-fine 2D-3D registration architecture to increase localization accuracy. We conduct extensive localization experiments on the KITTI Odometry and nuScenes datasets. The results demonstrate that I2PNet outperforms the state-of-the-art by a large margin. In addition, I2PNet has a higher efficiency than the previous works and can perform the localization in real-time. Moreover, we extend the application of I2PNet to the camera-LiDAR online calibration and demonstrate that I2PNet outperforms recent approaches on the online calibration task.

翻译：基于预先构建地图的机器人定位对于高精度导航和移动操作等多种任务至关重要。一种流行的机器人定位方法基于图像到点云的配准，该方法结合了光照不变的LiDAR地图构建与经济的图像定位。然而，近期图像到点云配准的研究要么将配准分解为独立模块，要么将点云投影为深度图像以实现RGB与深度图像的配准。本文提出I2PNet——一种新颖的端到端二维-三维配准网络。I2PNet利用具有统一目标的微分模块，直接实现原始三维点云与二维RGB图像的配准。我们提出用于微分二维-三维关联的二维-三维代价体模块，以桥接特征提取与位姿回归。该模块在针孔相机模型的内参无关归一化平面上隐式构建软点-像素对应关系。此外，我们引入离群掩膜预测模块，在位姿回归前过滤二维-三维关联中的离群点。进一步，我们提出粗到细的二维-三维配准架构以提升定位精度。我们在KITTI里程计和nuScenes数据集上进行了大量定位实验。结果表明，I2PNet以显著优势超越当前最优方法。同时，I2PNet较先前方法具有更高效率，可实现实时定位。此外，我们将I2PNet扩展应用于相机-LiDAR在线标定，并证明其在在线标定任务上优于近期方法。