Multi-instance scenes are especially challenging for end-to-end visuomotor (image-to-control) learning algorithms. "Pipeline" visual servo control algorithms use separate detection, selection and servo stages, allowing algorithms to focus on a single object instance during servo control. End-to-end systems do not have separate detection and selection stages and need to address the visual ambiguities introduced by the presence of arbitrary number of visually identical or similar objects during servo control. However, end-to-end schemes avoid embedding errors from detection and selection stages in the servo control behaviour, are more dynamically robust to changing scenes, and are algorithmically simpler. In this paper, we present a real-time end-to-end visuomotor learning algorithm for multi-instance reaching. The proposed algorithm uses a monocular RGB image and the manipulator's joint angles as the input to a light-weight fully-convolutional network (FCN) to generate control candidates. A key innovation of the proposed method is identifying the optimal control candidate by regressing a control-Lyapunov function (cLf) value. The multi-instance capability emerges naturally from the stability analysis associated with the cLf formulation. We demonstrate the proposed algorithm effectively reaching and grasping objects from different categories on a table-top amid other instances and distractors from an over-the-shoulder monocular RGB camera. The network is able to run up to approximately 160 fps during inference on one GTX 1080 Ti GPU.

翻译：多实例场景对端到端视觉运动（图像到控制）学习算法极具挑战性。传统流水线式视觉伺服控制算法采用分离的检测、选择与伺服阶段，使算法在伺服控制中能聚焦于单个目标实例。而端到端系统缺乏独立的检测与选择阶段，需应对伺服控制中数量不定的视觉相同或相似物体带来的视觉歧义。但端到端方案避免了检测与选择阶段产生的误差嵌入伺服控制行为，对动态场景变化更具鲁棒性，且算法实现更简洁。本文提出一种面向多实例抓取的实时端到端视觉运动学习算法。该算法以单目RGB图像和机械臂关节角为输入，通过轻量级全卷积网络（FCN）生成控制候选方案。其关键创新在于通过回归控制-李雅普诺夫函数（cLf）值识别最优控制候选方案，多实例能力自然源于cLf理论框架的稳定性分析。实验证明，该算法在桌面场景中，通过肩部单目RGB相机，可有效抓取包含干扰物的多类别目标实例。网络在单块GTX 1080 Ti GPU上推理速度可达约160帧/秒。