We study the problem of mobile manipulation using legged robots equipped with an arm, namely legged loco-manipulation. The robot legs, while usually utilized for mobility, offer an opportunity to amplify the manipulation capabilities by conducting whole-body control. That is, the robot can control the legs and the arm at the same time to extend its workspace. We propose a framework that can conduct the whole-body control autonomously with visual observations. Our approach, namely Visual Whole-Body Control(VBC), is composed of a low-level policy using all degrees of freedom to track the body velocities along with the end-effector position, and a high-level policy proposing the velocities and end-effector position based on visual inputs. We train both levels of policies in simulation and perform Sim2Real transfer for real robot deployment. We perform extensive experiments and show significant improvements over baselines in picking up diverse objects in different configurations (heights, locations, orientations) and environments.

翻译：我们研究了使用配备机械臂的腿足机器人进行移动操控的问题，即腿部移动操控。机器人腿部虽通常用于机动性，但可通过全身控制来增强操控能力——即机器人可同时控制腿部与机械臂以扩展其工作空间。本文提出一种基于视觉观测自主执行全身控制的框架。我们的方法——视觉全身控制（Visual Whole-Body Control, VBC）——包含两个层级：底层策略利用所有自由度跟踪身体速度与末端执行器位置，高层策略基于视觉输入生成速度与末端执行器目标。我们在仿真环境中训练两个层级的策略，并通过仿真到现实迁移实现真实机器人部署。通过大量实验，本方法在不同配置（高度、位置、朝向）与环境中的多样物体拾取任务上，相较于基线方法展现出显著性能提升。