Recent advancements in robotics have enabled robots to navigate complex scenes or manipulate diverse objects independently. However, robots are still impotent in many household tasks requiring coordinated behaviors such as opening doors. The factorization of navigation and manipulation, while effective for some tasks, fails in scenarios requiring coordinated actions. To address this challenge, we introduce, HarmonicMM, an end-to-end learning method that optimizes both navigation and manipulation, showing notable improvement over existing techniques in everyday tasks. This approach is validated in simulated and real-world environments and adapts to novel unseen settings without additional tuning. Our contributions include a new benchmark for mobile manipulation and the successful deployment with only RGB visual observation in a real unseen apartment, demonstrating the potential for practical indoor robot deployment in daily life. More results are on our project site: https://rchalyang.github.io/HarmonicMM/

翻译：机器人学的最新进展使得机器人能够独立地在复杂场景中导航或操作多种物体。然而，在许多需要协调行为（例如开门）的家庭任务中，机器人仍然无能为力。将导航与操作分离的方法虽然对某些任务有效，但在需要协调动作的场景中却行不通。为了应对这一挑战，我们提出了HarmonicMM，一种端到端的学习方法，它同时优化导航和操作，在日常任务中显示出相较于现有技术的显著改进。该方法在模拟和真实世界环境中得到了验证，并且能够适应未见过的全新场景而无需额外调整。我们的贡献包括一个新的移动操作基准，以及仅使用RGB视觉观测在一个真实、未见过的公寓中成功部署，展示了实际室内机器人在日常生活中部署的潜力。更多结果请见我们的项目网站：https://rchalyang.github.io/HarmonicMM/