The tie-knotting task is highly challenging due to the tie's high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate a sequence of tie's meshes from the demonstration video. With these estimated meshes used as subgoals, we first learn a teacher policy using privileged information. Then, we learn a student policy with point cloud observation by imitating teacher policy. Lastly, our pipeline learns a residual policy when the learned policy is applied to real-world execution, mitigating the Sim2Real gap. We demonstrate the effectiveness of TieBot in simulation and the real world. In the real-world experiment, a dual-arm robot successfully knots a tie, achieving 50% success rate among 10 trials. Videos can be found https://tiebots.github.io/.

翻译：系领带任务因领带的高度形变和长时程操作动作而极具挑战性。本研究提出TieBot，一种基于真实-仿真-真实范式的视觉演示学习系统，使机器人能够学习系领带。我们引入分层特征匹配方法，从演示视频中估计领带网格的时序序列。以这些估计网格作为子目标，我们首先利用特权信息学习教师策略。随后，通过模仿教师策略，基于点云观测学习学生策略。最后，当习得策略应用于真实世界执行时，我们的流程会学习残差策略以缓解仿真到现实的差距。我们在仿真环境与真实世界中验证了TieBot的有效性。真实世界实验中，双机械臂机器人成功完成系领带任务，在10次尝试中达到50%的成功率。演示视频可见 https://tiebots.github.io/。