Despite significant progress in robotic systems for operation within human-centric environments, existing models still heavily rely on explicit human commands to identify and manipulate specific objects. This limits their effectiveness in environments where understanding and acting on implicit human intentions are crucial. In this study, we introduce a novel task: reasoning grasping, where robots need to generate grasp poses based on indirect verbal instructions or intentions. To accomplish this, we propose an end-to-end reasoning grasping model that integrates a multi-modal Large Language Model (LLM) with a vision-based robotic grasping framework. In addition, we present the first reasoning grasping benchmark dataset generated from the GraspNet-1 billion, incorporating implicit instructions for object-level and part-level grasping, and this dataset will soon be available for public access. Our results show that directly integrating CLIP or LLaVA with the grasp detection model performs poorly on the challenging reasoning grasping tasks, while our proposed model demonstrates significantly enhanced performance both in the reasoning grasping benchmark and real-world experiments.

翻译：尽管机器人系统在人类中心环境中的操作取得了显著进展，现有模型仍严重依赖明确的人类指令来识别和操控特定物体。这限制了它们在理解并响应人类隐含意图至关重要的环境中的效能。本研究提出一项新任务：推理式抓取，即机器人需基于间接言语指令或意图生成抓取姿态。为此，我们提出一种端到端的推理式抓取模型，该模型将多模态大语言模型（LLM）与基于视觉的机器人抓取框架相结合。此外，我们发布了首个基于GraspNet-10亿生成的推理式抓取基准数据集，其中包含面向物体级和部件级抓取的隐含指令，该数据集即将公开可用。结果表明，直接将CLIP或LLaVA与抓取检测模型集成在具有挑战性的推理式抓取任务中表现不佳，而我们所提出的模型在推理式抓取基准测试和真实世界实验中均展现出显著提升的性能。