Knowledge of the 6D pose of an object can benefit in-hand object manipulation. In-hand 6D object pose estimation is challenging because of heavy occlusion produced by the robot's grippers, which can have an adverse effect on methods that rely on vision data only. Many robots are equipped with tactile sensors at their fingertips that could be used to complement vision data. In this paper, we present a method that uses both tactile and vision data to estimate the pose of an object grasped in a robot's hand. To address challenges like lack of standard representation for tactile data and sensor fusion, we propose the use of point clouds to represent object surfaces in contact with the tactile sensor and present a network architecture based on pixel-wise dense fusion. We also extend NVIDIA's Deep Learning Dataset Synthesizer to produce synthetic photo-realistic vision data and corresponding tactile point clouds. Results suggest that using tactile data in addition to vision data improves the 6D pose estimate, and our network generalizes successfully from synthetic training to real physical robots.

翻译：掌握物体的六维位姿信息有助于实现精细的在手物体操控。在手六维物体位姿估计面临的主要挑战在于机器人夹爪产生的严重遮挡，这对仅依赖视觉数据的方法会产生不利影响。许多机器人的指尖配备有触觉传感器，这些数据可用于补充视觉信息。本文提出一种同时利用触觉与视觉数据来估计机器人手中抓握物体位姿的方法。针对触觉数据缺乏标准表示形式及传感器融合等挑战，我们提出使用点云来表示与触觉传感器接触的物体表面，并构建了一种基于像素级密集融合的网络架构。同时，我们扩展了NVIDIA深度学习数据集合成器，用以生成合成逼真视觉数据及对应的触觉点云。实验结果表明，在视觉数据基础上引入触觉数据能够提升六维位姿估计精度，且我们的网络能够成功实现从合成训练到真实物理机器人的泛化。