Robots are finding wider adoption in human environments, increasing the need for natural human-robot interaction. However, understanding a natural language command requires the robot to infer the intended task and how to decompose it into executable actions, and to ground those actions in the robot's knowledge of the environment, including relevant objects, agents, and locations. This challenge can be addressed by combining the capabilities of Large language models (LLMs) to understand natural language with 3D scene graphs (3DSGs) for grounding inferred actions in a semantic representation of the environment. However, many 3DSGs lack explicit spatial relations between objects, even though humans often rely on these relations to describe an environment. This paper investigates whether incorporating open- or closed-vocabulary spatial relations into 3DSGs can improve the ability of LLMs to interpret natural language commands. To address this, we propose an LLM-based pipeline for target object grounding from open-vocabulary language commands and a vision language model (VLM)-based pipeline to add open-vocabulary spatial edges to 3DSGs from images captured while mapping. Finally, two LLMs are evaluated in a study assessing their performance on the downstream task of target object grounding. Our study demonstrates that explicit spatial relations improve the ability of LLMs to ground objects. Moreover, open-vocabulary relation generation with VLMs proves feasible from robot-captured images, but their advantage over closed-vocabulary relations is found to be limited.

翻译：随着机器人在人类环境中的应用日益广泛，自然的人机交互需求日益增长。然而，理解自然语言指令要求机器人推断预期任务、将其分解为可执行动作，并将这些动作在机器人对环境（包括相关物体、智能体和位置）的认知中进行定位。这一挑战可以通过结合大型语言模型（LLMs）理解自然语言的能力与三维场景图（3DSGs）在环境语义表征中对推断动作进行定位的能力来解决。然而，许多3DSGs缺乏物体间显式的空间关系，而人类在描述环境时常常依赖这些关系。本文研究了将开放或封闭词汇的空间关系纳入3DSGs是否能提升LLMs解释自然语言指令的能力。为此，我们提出了一种基于LLM的流程，用于从开放词汇语言指令中定位目标物体；以及一种基于视觉语言模型（VLM）的流程，用于通过建图过程中捕获的图像为3DSGs添加开放词汇的空间边。最后，通过一项研究评估了两个LLMs在目标物体定位下游任务中的性能。我们的研究表明，显式空间关系能有效提升LLMs的物体定位能力。此外，利用VLMs从机器人捕获图像生成开放词汇关系被证明是可行的，但其相较于封闭词汇关系的优势较为有限。