We introduce UMI-on-Legs, a new framework that combines real-world and simulation data for quadruped manipulation systems. We scale task-centric data collection in the real world using a hand-held gripper (UMI), providing a cheap way to demonstrate task-relevant manipulation skills without a robot. Simultaneously, we scale robot-centric data in simulation by training whole-body controller for task-tracking without task simulation setups. The interface between these two policies is end-effector trajectories in the task frame, inferred by the manipulation policy and passed to the whole-body controller for tracking. We evaluate UMI-on-Legs on prehensile, non-prehensile, and dynamic manipulation tasks, and report over 70% success rate on all tasks. Lastly, we demonstrate the zero-shot cross-embodiment deployment of a pre-trained manipulation policy checkpoint from prior work, originally intended for a fixed-base robot arm, on our quadruped system. We believe this framework provides a scalable path towards learning expressive manipulation skills on dynamic robot embodiments. Please checkout our website for robot videos, code, and data: https://umi-on-legs.github.io

翻译：我们提出了UMI-on-Legs，一个结合真实世界与仿真数据、面向四足机器人操作系统的全新框架。我们利用手持夹爪（UMI）在真实世界中规模化收集以任务为中心的数据，提供了一种无需机器人即可演示任务相关操作技能的廉价方法。同时，我们在仿真中通过训练全身控制器来跟踪任务轨迹（无需搭建任务仿真环境），从而规模化获取以机器人为中心的数据。这两个策略之间的接口是任务坐标系中的末端执行器轨迹，由操作策略推断生成并传递给全身控制器进行跟踪。我们在抓取、非抓取及动态操作任务上评估了UMI-on-Legs，所有任务均报告超过70%的成功率。最后，我们展示了预训练操作策略检查点的零样本跨本体部署能力：将先前工作中为固定基座机械臂设计的策略直接应用于我们的四足机器人系统。我们相信该框架为在动态机器人本体上学习丰富操作技能提供了一条可扩展的路径。请访问我们的网站查看机器人视频、代码与数据：https://umi-on-legs.github.io