Language-Conditioned Robotic Grasping (LCRG) aims to develop robots that comprehend and grasp objects based on natural language instructions. While the ability to understand personal objects like my wallet facilitates more natural interaction with human users, current LCRG systems only allow generic language instructions, e.g., the black-colored wallet next to the laptop. To this end, we introduce a task scenario GraspMine alongside a novel dataset aimed at pinpointing and grasping personal objects given personal indicators via learning from a single human-robot interaction, rather than a large labeled dataset. Our proposed method, Personalized Grasping Agent (PGA), addresses GraspMine by leveraging the unlabeled image data of the user's environment, called Reminiscence. Specifically, PGA acquires personal object information by a user presenting a personal object with its associated indicator, followed by PGA inspecting the object by rotating it. Based on the acquired information, PGA pseudo-labels objects in the Reminiscence by our proposed label propagation algorithm. Harnessing the information acquired from the interactions and the pseudo-labeled objects in the Reminiscence, PGA adapts the object grounding model to grasp personal objects. This results in significant efficiency while previous LCRG systems rely on resource-intensive human annotations -- necessitating hundreds of labeled data to learn my wallet. Moreover, PGA outperforms baseline methods across all metrics and even shows comparable performance compared to the fully-supervised method, which learns from 9k annotated data samples. We further validate PGA's real-world applicability by employing a physical robot to execute GrsapMine. Code and data are publicly available at https://github.com/JHKim-snu/PGA.

翻译：摘要：语言条件化机器人抓取（LCRG）旨在开发能够基于自然语言指令理解并抓取物体的机器人。尽管识别个人物品（如“我的钱包”）的能力有助于实现更自然的人机交互，但现有LCRG系统仅支持通用语言指令（例如，“笔记本电脑旁边的黑色钱包”）。为此，我们提出了一项新任务场景GraspMine及其配套数据集，旨在通过单次人机交互学习而非大规模标注数据集，基于个人指示符定位并抓取个人物品。所提出的方法PGA（个性化抓取代理）通过利用用户环境中未标注的图像数据（称为“回忆”Reminiscence）来解决GraspMine任务。具体而言，PGA由用户展示个人物品及其关联指示符来获取个人物品信息，随后通过旋转物品对其进行检测。基于获取的信息，PGA通过我们提出的标签传播算法为回忆中的物体生成伪标签。结合交互中获取的信息与回忆中的伪标签物体，PGA自适应调整物体接地模型以实现个人物品抓取。这一方法实现了显著的高效性，而现有LCRG系统依赖资源密集型的人工标注——例如需要数百个标注数据来学习“我的钱包”。此外，PGA在所有指标上均优于基线方法，甚至与使用9000个标注数据样本的全监督方法性能相当。我们进一步通过物理机器人执行GraspMine任务验证了PGA的实际应用可行性。代码与数据已开源：https://github.com/JHKim-snu/PGA。