We focus on the task of language-conditioned grasping in clutter, in which a robot is supposed to grasp the target object based on a language instruction. Previous works separately conduct visual grounding to localize the target object, and generate a grasp for that object. However, these works require object labels or visual attributes for grounding, which calls for handcrafted rules in planner and restricts the range of language instructions. In this paper, we propose to jointly model vision, language and action with object-centric representation. Our method is applicable under more flexible language instructions, and not limited by visual grounding error. Besides, by utilizing the powerful priors from the pre-trained multi-modal model and grasp model, sample efficiency is effectively improved and the sim2real problem is relived without additional data for transfer. A series of experiments carried out in simulation and real world indicate that our method can achieve better task success rate by less times of motion under more flexible language instructions. Moreover, our method is capable of generalizing better to scenarios with unseen objects and language instructions.

翻译：我们聚焦于语言条件引导下的杂乱环境抓取任务，即机器人需依据语言指令抓取目标物体。以往研究将视觉定位（定位目标物体）与抓取生成分离进行。然而，这些方法需要物体标签或视觉属性进行定位，这要求规划器中采用人工设定规则，且限制了语言指令的多样性。本文提出通过以物体为中心的表示联合建模视觉、语言与动作。我们的方法适用于更灵活的语言指令，且不受视觉定位误差限制。此外，通过利用预训练多模态模型与抓取模型的强大先验知识，有效提升了样本效率，并缓解了仿真到现实迁移问题，无需额外迁移数据。在仿真与真实环境中进行的一系列实验表明，我们的方法能在更灵活的语言指令下，通过更少的运动次数实现更高的任务成功率。同时，该方法在应对未见物体与未见语言指令的场景时展现出更强的泛化能力。