Despite significant progress in robotic systems for operation within human-centric environments, existing models still heavily rely on explicit human commands to identify and manipulate specific objects. This limits their effectiveness in environments where understanding and acting on implicit human intentions are crucial. In this study, we introduce a novel task: reasoning grasping, where robots need to generate grasp poses based on indirect verbal instructions or intentions. To accomplish this, we propose an end-to-end reasoning grasping model that integrates a multi-modal Large Language Model (LLM) with a vision-based robotic grasping framework. In addition, we present the first reasoning grasping benchmark dataset generated from the GraspNet-1 billion, incorporating implicit instructions for object-level and part-level grasping, and this dataset will soon be available for public access. Our results show that directly integrating CLIP or LLaVA with the grasp detection model performs poorly on the challenging reasoning grasping tasks, while our proposed model demonstrates significantly enhanced performance both in the reasoning grasping benchmark and real-world experiments.

翻译：尽管面向人类环境操作的机器人系统取得了显著进展，但现有模型仍严重依赖明确的人类指令来识别和操控特定物体。这限制了其在需要理解并执行隐含人类意图的关键场景中的有效性。本研究提出了一项新任务：推理抓取，即机器人需根据间接语言指令或隐含意图生成抓取姿态。为实现该目标，我们提出了一种端到端的推理抓取模型，该模型将多模态大语言模型（LLM）与基于视觉的机器人抓取框架相结合。此外，我们首次发布了基于GraspNet-10亿数据集生成的推理抓取基准测试数据集，该数据集包含面向物体级与部件级抓取的隐含指令，并即将公开开放。实验结果表明，直接将CLIP或LLaVA与抓取检测模型集成在具有挑战性的推理抓取任务中表现较差，而我们所提出的模型在推理抓取基准测试及真实世界实验中均展现出显著增强的性能。