Solving the camera-to-robot pose is a fundamental requirement for vision-based robot control, and is a process that takes considerable effort and cares to make accurate. Traditional approaches require modification of the robot via markers, and subsequent deep learning approaches enabled markerless feature extraction. Mainstream deep learning methods only use synthetic data and rely on Domain Randomization to fill the sim-to-real gap, because acquiring the 3D annotation is labor-intensive. In this work, we go beyond the limitation of 3D annotations for real-world data. We propose an end-to-end pose estimation framework that is capable of online camera-to-robot calibration and a self-supervised training method to scale the training to unlabeled real-world data. Our framework combines deep learning and geometric vision for solving the robot pose, and the pipeline is fully differentiable. To train the Camera-to-Robot Pose Estimation Network (CtRNet), we leverage foreground segmentation and differentiable rendering for image-level self-supervision. The pose prediction is visualized through a renderer and the image loss with the input image is back-propagated to train the neural network. Our experimental results on two public real datasets confirm the effectiveness of our approach over existing works. We also integrate our framework into a visual servoing system to demonstrate the promise of real-time precise robot pose estimation for automation tasks.

翻译：求解相机到机器人的位姿是基于视觉的机器人控制的基本要求，这一过程需要大量精力和细致操作才能实现高精度。传统方法需要对机器人进行标记改造，而后续的深度学习方法实现了无标记特征提取。主流深度学习方法仅使用合成数据，并依赖域随机化技术填补模拟到真实的差距，因为获取三维标注数据需要大量人力。在本工作中，我们突破了真实世界数据对三维标注的限制。我们提出了一种能够实现在线相机-机器人标定的端到端位姿估计框架，以及一种可扩展至无标签真实世界数据训练的自监督训练方法。该框架融合了深度学习与几何视觉方法解决机器人位姿问题，且整个流程完全可微。为了训练相机-机器人位姿估计网络（CtRNet），我们利用前景分割与可微渲染实现图像级自监督。通过渲染器可视化位姿预测结果，并将与输入图像之间的图像损失反向传播以训练神经网络。在两个公开真实数据集上的实验结果证实了我们的方法相对于现有工作的有效性。我们还将该框架集成到视觉伺服系统中，展示了实时高精度机器人位姿估计在自动化任务中的潜力。