Improving the generalization capabilities of general-purpose robotic manipulation agents in the real world has long been a significant challenge. Existing approaches often rely on collecting large-scale robotic data which is costly and time-consuming, such as the RT-1 dataset. However, due to insufficient diversity of data, these approaches typically suffer from limiting their capability in open-domain scenarios with new objects, and diverse environments. In this paper, we propose a novel paradigm that effectively leverages language grounded segmentation mask generated by Internet-scale foundation models, to address a wide range of pick-and-place robot manipulation tasks. By integrating the mask modality, which incorporates semantic, geometric, and temporal correlation priors derived from vision foundation models, into the end-to-end policy model, our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning, including new object instances, semantic categories, and unseen backgrounds. We first introduce a series of foundation models to ground natural language demands across multiple tasks. Secondly, we develop a two-stream 2D policy model based on imitation learning, which utilizes raw images, object masks, and robot proprioception to predict robot actions. Extensive real-world experiments conducted on a Franka Emika robot arm demonstrate the effectiveness of our proposed paradigm. Demos are shown in YouTube (https://www.youtube.com/watch?v=MAcUPFBfRIw ).

翻译：提升通用机器人操作代理在真实世界中的泛化能力一直是一个重大挑战。现有方法通常依赖于收集大规模机器人数据，如RT-1数据集，但此类数据获取成本高昂且耗时。然而，由于数据多样性不足，这些方法在开放域场景中面对新物体和多样化环境时往往能力受限。本文提出一种新颖范式，有效利用互联网规模基础模型生成的基于语言的分割掩码，以解决广泛的拾取与放置机器人操作任务。通过将源自视觉基础模型的语义、几何和时序相关先验的掩码模态集成到端到端策略模型中，我们的方法能够有效且鲁棒地感知物体姿态，并实现样本高效的泛化学习，包括新物体实例、语义类别和未见背景。我们首先引入一系列基础模型来跨多个任务对接自然语言需求；其次，基于模仿学习开发了一种双流二维策略模型，该模型利用原始图像、物体掩码和机器人本体感知来预测机器人动作。在Franka Emika机器人臂上开展的大量真实世界实验证明了我们提出范式的有效性。演示视频见YouTube (https://www.youtube.com/watch?v=MAcUPFBfRIw)。