The ability to handle objects in cluttered environment has been long anticipated by robotic community. However, most of works merely focus on manipulation instead of rendering hidden semantic information in cluttered objects. In this work, we introduce the scene graph for embodied exploration in cluttered scenarios to solve this problem. To validate our method in cluttered scenario, we adopt the Manipulation Question Answering (MQA) tasks as our test benchmark, which requires an embodied robot to have the active exploration ability and semantic understanding ability of vision and language.As a general solution framework to the task, we propose an imitation learning method to generate manipulations for exploration. Meanwhile, a VQA model based on dynamic scene graph is adopted to comprehend a series of RGB frames from wrist camera of manipulator along with every step of manipulation is conducted to answer questions in our framework.The experiments on of MQA dataset with different interaction requirements demonstrate that our proposed framework is effective for MQA task a representative of tasks in cluttered scenario.

翻译：在杂乱环境中处理物体的能力一直是机器人领域长期期望的目标。然而，现有工作大多仅关注操作本身，而非揭示杂乱物体中的隐含语义信息。为解决这一问题，本研究引入场景图实现杂乱场景下的具身探索。为验证所提方法在杂乱场景中的有效性，我们采用操作问答（Manipulation Question Answering，MQA）任务作为测试基准。该任务要求具身机器人具备主动探索能力以及视觉与语言的语义理解能力。作为该任务的通用解决方案框架，我们提出一种模仿学习方法以生成用于探索的操作动作。同时，基于动态场景图的视觉问答（VQA）模型被用于理解机械臂腕部相机采集的系列RGB帧——每一步操作执行后，该模型均对帧序列进行解析以回答相关问题。在不同交互需求的MQA数据集上的实验表明，所提框架对作为杂乱场景典型任务的MQA具有有效性。