We consider the problem of Embodied Question Answering (EQA), which refers to settings where an embodied agent such as a robot needs to actively explore an environment to gather information until it is confident about the answer to a question. In this work, we leverage the strong semantic reasoning capabilities of large vision-language models (VLMs) to efficiently explore and answer such questions. However, there are two main challenges when using VLMs in EQA: they do not have an internal memory for mapping the scene to be able to plan how to explore over time, and their confidence can be miscalibrated and can cause the robot to prematurely stop exploration or over-explore. We propose a method that first builds a semantic map of the scene based on depth information and via visual prompting of a VLM - leveraging its vast knowledge of relevant regions of the scene for exploration. Next, we use conformal prediction to calibrate the VLM's question answering confidence, allowing the robot to know when to stop exploration - leading to a more calibrated and efficient exploration strategy. To test our framework in simulation, we also contribute a new EQA dataset with diverse, realistic human-robot scenarios and scenes built upon the Habitat-Matterport 3D Research Dataset (HM3D). Both simulated and real robot experiments show our proposed approach improves the performance and efficiency over baselines that do no leverage VLM for exploration or do not calibrate its confidence. Webpage with experiment videos and code: https://explore-eqa.github.io/

翻译：本文研究具身问答问题，该场景要求具身智能体（如机器人）主动探索环境以收集信息，直至对问题答案具有足够置信度。本工作利用大规模视觉语言模型强大的语义推理能力，以高效探索并回答此类问题。然而，在具身问答中应用视觉语言模型面临两大挑战：其一，模型缺乏内部记忆机制来构建场景地图，难以实现时序探索规划；其二，模型的置信度可能存在校准偏差，可能导致机器人过早终止探索或过度探索。我们提出一种方法：首先基于深度信息并通过视觉语言模型的视觉提示构建场景语义地图——利用其关于场景相关区域的先验知识指导探索过程；其次，采用保形预测方法对视觉语言模型的问答置信度进行校准，使机器人能准确判断何时终止探索，从而形成更可靠高效的探索策略。为在仿真环境中测试框架，我们基于Habitat-Matterport 3D研究数据集构建了包含多样化真实人机交互场景的新型具身问答数据集。仿真与真实机器人实验均表明，相较于未利用视觉语言模型进行探索或未校准置信度的基线方法，我们提出的方案在性能与效率上均有显著提升。实验视频与代码网页：https://explore-eqa.github.io/