Enabling home-assistant robots to perceive and manipulate a diverse range of 3D objects based on human language instructions is a pivotal challenge. Prior research has predominantly focused on simplistic and task-oriented instructions, i.e., "Slide the top drawer open". However, many real-world tasks demand intricate multi-step reasoning, and without human instructions, these will become extremely difficult for robot manipulation. To address these challenges, we introduce a comprehensive benchmark, NrVLM, comprising 15 distinct manipulation tasks, containing over 4500 episodes meticulously annotated with fine-grained language instructions. We split the long-term task process into several steps, with each step having a natural language instruction. Moreover, we propose a novel learning framework that completes the manipulation task step-by-step according to the fine-grained instructions. Specifically, we first identify the instruction to execute, taking into account visual observations and the end-effector's current state. Subsequently, our approach facilitates explicit learning through action-prompts and perception-prompts to promote manipulation-aware cross-modality alignment. Leveraging both visual observations and linguistic guidance, our model outputs a sequence of actionable predictions for manipulation, including contact points and end-effector poses. We evaluate our method and baselines using the proposed benchmark NrVLM. The experimental results demonstrate the effectiveness of our approach. For additional details, please refer to https://sites.google.com/view/naturalvlm.

翻译：使家庭辅助机器人能够基于人类语言指令感知并操作多样化三维物体是一项关键挑战。先前研究主要关注简单且面向任务的指令，例如"拉开顶部抽屉"。然而，许多现实任务需要复杂的多步推理，若无人类指令指引，这些操作对机器人而言将极为困难。为解决上述挑战，我们提出了一个综合性基准测试NrVLM，涵盖15种不同的操作任务，包含4500余个经过细粒度语言指令精确标注的操作片段。我们将长期任务过程分解为若干步骤，并为每个步骤配以自然语言指令。此外，我们提出了一种新颖的学习框架，根据细粒度指令逐步完成操作任务。具体而言，我们首先结合视觉观察与末端执行器的当前状态，确定待执行的指令。随后，我们的方法通过动作提示与感知提示实现显式学习，以促进面向操作感知的跨模态对齐。模型融合视觉观察与语言引导，输出一系列可操作的操作预测结果，包括接触点与末端执行器姿态。我们利用所提出的NrVLM基准测试对方法及基线模型进行了评估，实验结果验证了该方法的有效性。更多详情请参见https://sites.google.com/view/naturalvlm。