Recently, several works achieve end-to-end visual servoing (VS) for robotic manipulation by replacing traditional controller with differentiable neural networks, but lose the ability to servo arbitrary desired poses. This letter proposes a differentiable architecture for arbitrary pose servoing: a hyper-network based neural controller (HPN-NC). To achieve this, HPN-NC consists of a hyper net and a low-level controller, where the hyper net learns to generate the parameters of the low-level controller and the controller uses the 2D keypoints error for control like traditional image-based visual servoing (IBVS). HPN-NC can complete 6 degree of freedom visual servoing with large initial offset. Taking advantage of the fully differentiable nature of HPN-NC, we provide a three-stage training procedure to servo real world objects. With self-supervised end-to-end training, the performance of the integrated model can be further improved in unseen scenes and the amount of manual annotations can be significantly reduced.

翻译：近期，多项研究通过用可微分神经网络替代传统控制器实现机器人操作中的端到端视觉伺服，但丧失了伺服任意期望位姿的能力。本文提出一种面向任意位姿伺服的可微分架构：基于超网络的神经控制器（HPN-NC）。为此，HPN-NC由超网络和底层控制器构成，其中超网络学习生成底层控制器的参数，而控制器则像传统基于图像的视觉伺服（IBVS）一样利用二维关键点误差进行控制。HPN-NC能够完成具有大初始偏移的六自由度视觉伺服任务。借助HPN-NC完全可微分的特性，我们提出一个三阶段训练流程以伺服真实世界物体。通过自监督端到端训练，集成模型在未见场景中的性能可进一步提升，同时显著减少人工标注量。