The objective of this work is to enable manipulation tasks with respect to the 6D pose of a dynamically moving object using a camera mounted on a robot. Examples include maintaining a constant relative 6D pose of the robot arm with respect to the object, grasping the dynamically moving object, or co-manipulating the object together with a human. Fast and accurate 6D pose estimation is crucial to achieve smooth and stable robot control in such situations. The contributions of this work are three fold. First, we propose a new visual perception module that asynchronously combines accurate learning-based 6D object pose localizer and a high-rate model-based 6D pose tracker. The outcome is a low-latency accurate and temporally consistent 6D object pose estimation from the input video stream at up to 120 Hz. Second, we develop a visually guided robot arm controller that combines the new visual perception module with a torque-based model predictive control algorithm. Asynchronous combination of the visual and robot proprioception signals at their corresponding frequencies results in stable and robust 6D object pose guided robot arm control. Third, we experimentally validate the proposed approach on a challenging 6D pose estimation benchmark and demonstrate 6D object pose-guided control with dynamically moving objects on a real 7 DoF Franka Emika Panda robot.

翻译：本文旨在利用机器人搭载的摄像头，实现对动态运动物体的六维位姿进行操控任务。例如包括保持机器人手臂相对于物体的恒定相对六维位姿、抓取动态运动物体，或与人类共同操控该物体。在此类场景中，快速且准确的六维位姿估计对实现平滑稳定的机器人控制至关重要。本文贡献分为三部分：首先，提出一种新型视觉感知模块，该模块异步结合了基于学习的高精度六维物体位姿定位器与高速率基于模型的六维位姿跟踪器，从而从输入视频流中以高达120Hz的频率输出低延迟、精确且时间一致的六维物体位姿估计；其次，开发一种视觉引导的机器人手臂控制器，将新型视觉感知模块与基于力矩的模型预测控制算法相结合，通过异步融合视觉信号与机器人本体感知信号（分别以对应频率运行），实现稳定鲁棒的六维物体位姿引导的机器人手臂控制；最后，在具有挑战性的六维位姿估计基准上对所提方法进行实验验证，并在真实7自由度Franka Emika Panda机器人上展示了针对动态运动物体的六维物体位姿引导控制。