The commonly adopted detect-then-match approach to registration finds difficulties in the cross-modality cases due to the incompatible keypoint detection and inconsistent feature description. We propose, 2D3D-MATR, a detection-free method for accurate and robust registration between images and point clouds. Our method adopts a coarse-to-fine pipeline where it first computes coarse correspondences between downsampled patches of the input image and the point cloud and then extends them to form dense correspondences between pixels and points within the patch region. The coarse-level patch matching is based on transformer which jointly learns global contextual constraints with self-attention and cross-modality correlations with cross-attention. To resolve the scale ambiguity in patch matching, we construct a multi-scale pyramid for each image patch and learn to find for each point patch the best matching image patch at a proper resolution level. Extensive experiments on two public benchmarks demonstrate that 2D3D-MATR outperforms the previous state-of-the-art P2-Net by around $20$ percentage points on inlier ratio and over $10$ points on registration recall. Our code and models are available at \url{https://github.com/minhaolee/2D3DMATR}.

翻译：[translated abstract in Chinese] 常用的“先检测后匹配”式配准方法在跨模态场景中因关键点检测不兼容及特征描述不一致而面临困难。本文提出2D3D-MATR——一种无检测器方法，实现图像与点云间的精确鲁棒配准。该方法采用由粗到细的流水线：首先计算输入图像下采样块与点云之间的粗粒度对应关系，随后将其扩展为像素与点云块内点之间的稠密对应。粗粒度块匹配基于Transformer架构，通过自注意力机制联合学习全局上下文约束，利用交叉注意力机制建模跨模态相关性。为解决块匹配中的尺度模糊问题，我们为每个图像块构建多尺度金字塔，并学习为每个点块在合适分辨率层级寻找最优匹配图像块。在两个公开基准上的大量实验表明，相比先前最优方法P2-Net，2D3D-MATR的内点率提升约20个百分点，配准召回率提升超过10个百分点。代码与模型已开源：\url{https://github.com/minhaolee/2D3DMATR}。