Robots operating in human-centric environments require the integration of visual grounding and grasping capabilities to effectively manipulate objects based on user instructions. This work focuses on the task of referring grasp synthesis, which predicts a grasp pose for an object referred through natural language in cluttered scenes. Existing approaches often employ multi-stage pipelines that first segment the referred object and then propose a suitable grasp, and are evaluated in private datasets or simulators that do not capture the complexity of natural indoor scenes. To address these limitations, we develop a challenging benchmark based on cluttered indoor scenes from OCID dataset, for which we generate referring expressions and connect them with 4-DoF grasp poses. Further, we propose a novel end-to-end model (CROG) that leverages the visual grounding capabilities of CLIP to learn grasp synthesis directly from image-text pairs. Our results show that vanilla integration of CLIP with pretrained models transfers poorly in our challenging benchmark, while CROG achieves significant improvements both in terms of grounding and grasping. Extensive robot experiments in both simulation and hardware demonstrate the effectiveness of our approach in challenging interactive object grasping scenarios that include clutter.

翻译：在人类为中心的环境中运行的机器人需要整合视觉定位与抓取能力，以便根据用户指令有效操作物体。本研究聚焦于指代抓取合成任务，即在杂散场景中预测由自然语言指代物体的抓取姿态。现有方法通常采用多阶段流程，先分割出指代物体，再提出合适的抓取方案，且多在无法捕捉自然室内场景复杂性的私有数据集或模拟器中进行评估。为解决这些局限，我们基于OCID数据集中的杂散室内场景开发了一个具有挑战性的基准测试，为其生成指代表达并将其与4自由度抓取姿态关联。进一步，我们提出了一种新型端到端模型（CROG），该模型利用CLIP的视觉定位能力直接从图像-文本对中学习抓取合成。实验结果表明，在具有挑战性的基准测试中，将CLIP与预训练模型简单集成的方法迁移效果较差，而CROG在定位和抓取两方面均实现显著提升。在仿真平台与硬件设备上进行的广泛机器人实验证明，我们的方法在包含杂散物的交互式物体抓取场景中具有有效性。