The ability to detect and analyze failed executions automatically is crucial for an explainable and robust robotic system. Recently, Large Language Models (LLMs) have demonstrated strong reasoning abilities on textual inputs. To leverage the power of LLMs for robot failure explanation, we introduce REFLECT, a framework which queries LLM for failure reasoning based on a hierarchical summary of robot past experiences generated from multisensory observations. The failure explanation can further guide a language-based planner to correct the failure and complete the task. To systematically evaluate the framework, we create the RoboFail dataset with a variety of tasks and failure scenarios. We demonstrate that the LLM-based framework is able to generate informative failure explanations that assist successful correction planning.

翻译：自动检测和分析执行失败的能力对于构建可解释且鲁棒的机器人系统至关重要。近年来，大语言模型（LLMs）在文本输入上展现出强大的推理能力。为利用LLMs进行机器人故障解释，我们提出REFLECT框架——该框架基于多模态感知生成的机器人历史经验分层总结，通过查询LLM进行故障推理。故障解释可进一步指导基于语言的规划器修正故障并完成任务。为系统评估该框架，我们创建了包含多种任务与故障场景的RoboFail数据集。实验证明，该基于LLM的框架能够生成具有信息量的故障解释，从而有效辅助修正规划。