Locating 3D objects from a single RGB image via Perspective-n-Point (PnP) is a long-standing problem in computer vision. Driven by end-to-end deep learning, recent studies suggest interpreting PnP as a differentiable layer, allowing for partial learning of 2D-3D point correspondences by backpropagating the gradients of pose loss. Yet, learning the entire correspondences from scratch is highly challenging, particularly for ambiguous pose solutions, where the globally optimal pose is theoretically non-differentiable w.r.t. the points. In this paper, we propose the EPro-PnP, a probabilistic PnP layer for general end-to-end pose estimation, which outputs a distribution of pose with differentiable probability density on the SE(3) manifold. The 2D-3D coordinates and corresponding weights are treated as intermediate variables learned by minimizing the KL divergence between the predicted and target pose distribution. The underlying principle generalizes previous approaches, and resembles the attention mechanism. EPro-PnP can enhance existing correspondence networks, closing the gap between PnP-based method and the task-specific leaders on the LineMOD 6DoF pose estimation benchmark. Furthermore, EPro-PnP helps to explore new possibilities of network design, as we demonstrate a novel deformable correspondence network with the state-of-the-art pose accuracy on the nuScenes 3D object detection benchmark. Our code is available at https://github.com/tjiiv-cprg/EPro-PnP-v2.

翻译：通过透视n点法（PnP）从单张RGB图像中定位3D物体是计算机视觉领域的一个长期难题。受端到端深度学习的驱动，近期研究建议将PnP解释为可微分层，通过反向传播姿态损失的梯度实现2D-3D点对应关系的部分学习。然而，从零开始学习全部对应关系极具挑战性，尤其在存在歧义姿态解的情况下——此时全局最优姿态对点坐标在理论上不可微。本文提出EPro-PnP，一种用于通用端到端姿态估计的概率化PnP层，它在SE(3)流形上输出具有可微概率密度的姿态分布。2D-3D坐标及其对应权重作为中间变量，通过最小化预测姿态分布与目标姿态分布之间的KL散度进行学习。该原理既泛化了先前方法，又类似于注意力机制。EPro-PnP可增强现有对应关系网络，在LineMOD 6DoF姿态估计基准上将基于PnP的方法与任务专用领先方法之间的差距缩小。此外，EPro-PnP有助于探索网络设计的新可能性——我们展示了一种新型可变形对应关系网络，在nuScenes 3D目标检测基准上实现了最先进的姿态精度。我们的代码开源在https://github.com/tjiiv-cprg/EPro-PnP-v2。