Recently, the advent of pre-trained large-scale language models (LLMs) like ChatGPT and GPT-4 have significantly advanced the machine's natural language understanding capabilities. This breakthrough has allowed us to seamlessly integrate these open-source LLMs into a unified robot simulator environment to help robots accurately understand and execute human natural language instructions. To this end, in this work, we introduce a realistic robotic manipulation simulator and build a Robotic Manipulation with Progressive Reasoning Tasks (RM-PRT) benchmark on this basis. Specifically, the RM-PRT benchmark builds a new high-fidelity digital twin scene based on Unreal Engine 5, which includes 782 categories, 2023 objects, and 15K natural language instructions generated by ChatGPT for a detailed evaluation of robot manipulation. We propose a general pipeline for the RM-PRT benchmark that takes as input multimodal prompts containing natural language instructions and automatically outputs actions containing the movement and position transitions. We set four natural language understanding tasks with progressive reasoning levels and evaluate the robot's ability to understand natural language instructions in two modes of adsorption and grasping. In addition, we also conduct a comprehensive analysis and comparison of the differences and advantages of 10 different LLMs in instruction understanding and generation quality. We hope the new simulator and benchmark will facilitate future research on language-guided robotic manipulation. Project website: https://necolizer.github.io/RM-PRT/ .

翻译：近期，预训练大规模语言模型（如ChatGPT和GPT-4）的出现显著提升了机器的自然语言理解能力。这一突破使我们能够将这些开源语言模型无缝集成到统一的机器人模拟器环境中，帮助机器人准确理解并执行人类的自然语言指令。为此，本研究引入了一个真实的机器人操控模拟器，并在此基础上构建了面向渐进式推理任务的机器人操控基准（RM-PRT）。具体而言，RM-PRT基准基于虚幻引擎5构建了新型高保真数字孪生场景，包含782个类别、2023个物体以及由ChatGPT生成的15K条自然语言指令，用于对机器人操控进行精细化评估。我们提出了一套通用流程：该流程以包含自然语言指令的多模态提示为输入，自动输出包含运动与位置变化的动作。我们设置了四个具有渐进式推理层级的自然语言理解任务，并在吸附与抓取两种模式下评估机器人理解自然语言指令的能力。此外，我们还对10种不同语言模型在指令理解与生成质量方面的差异与优势进行了全面分析与比较。我们期望这一新型模拟器与基准能够推动基于语言的机器人操控研究。项目网站：https://necolizer.github.io/RM-PRT/。