In this paper, we propose LAN-grasp, a novel approach towards more appropriate semantic grasping. We use foundation models to provide the robot with a deeper understanding of the objects, the right place to grasp an object, or even the parts to avoid. This allows our robot to grasp and utilize objects in a more meaningful and safe manner. We leverage the combination of a Large Language Model, a Vision Language Model, and a traditional grasp planner to generate grasps demonstrating a deeper semantic understanding of the objects. We first prompt the Large Language Model about which object part is appropriate for grasping. Next, the Vision Language Model identifies the corresponding part in the object image. Finally, we generate grasp proposals in the region proposed by the Vision Language Model. Building on foundation models provides us with a zero-shot grasp method that can handle a wide range of objects without the need for further training or fine-tuning. We evaluated our method in real-world experiments on a custom object data set. We present the results of a survey that asks the participants to choose an object part appropriate for grasping. The results show that the grasps generated by our method are consistently ranked higher by the participants than those generated by a conventional grasping planner and a recent semantic grasping approach.

翻译：摘要：本文提出LAN-grasp，一种面向更优语义抓取任务的新颖方法。我们利用基础模型赋予机器人对物体的深度理解能力，使其能精准识别抓取部位、明确应避开的区域，从而实现更具意义性和安全性的物体抓取与操作。通过融合大型语言模型、视觉语言模型与传统抓取规划器，我们生成的抓取策略展现出对物体的深层语义理解。具体流程为：首先向大型语言模型查询物体适宜抓取的部位，随后视觉语言模型在物体图像中定位该部位，最终在视觉语言模型建议区域生成抓取候选方案。基于基础模型构建的方法具备零样本泛化能力，无需额外训练或微调即可处理多样化的物体对象。我们在自建物体数据集上开展了真实环境实验评估，并通过问卷调研要求参与者选择适宜抓取部位。结果表明，与传统抓取规划器及近期语义抓取方法相比，我们的方法生成的抓取方案始终获得参与者更高评分。