Classical planning systems have shown great advances in utilizing rule-based human knowledge to compute accurate plans for service robots, but they face challenges due to the strong assumptions of perfect perception and action executions. To tackle these challenges, one solution is to connect the symbolic states and actions generated by classical planners to the robot's sensory observations, thus closing the perception-action loop. This research proposes a visually-grounded planning framework, named TPVQA, which leverages Vision-Language Models (VLMs) to detect action failures and verify action affordances towards enabling successful plan execution. Results from quantitative experiments show that TPVQA surpasses competitive baselines from previous studies in task completion rate.

翻译：经典规划系统在利用基于规则的人类知识为服务机器人计算精确规划方面取得了显著进展，但由于其依赖于完美感知和动作执行的强假设，面临诸多挑战。为解决这些问题，一种方案是将经典规划器生成的符号状态和动作与机器人的感知观测相连接，从而闭环感知-动作循环。本研究提出了一种名为TPVQA的视觉锚定规划框架，该框架利用视觉-语言模型（VLM）检测动作失败并验证动作可行性，以实现成功执行规划。定量实验结果表明，TPVQA在任务完成率上超过了以往研究中的竞争基线方法。