We present ConVOI, a novel method for autonomous robot navigation in real-world indoor and outdoor environments using Vision Language Models (VLMs). We employ VLMs in two ways: first, we leverage their zero-shot image classification capability to identify the context or scenario (e.g., indoor corridor, outdoor terrain, crosswalk, etc) of the robot's surroundings, and formulate context-based navigation behaviors as simple text prompts (e.g. ``stay on the pavement"). Second, we utilize their state-of-the-art semantic understanding and logical reasoning capabilities to compute a suitable trajectory given the identified context. To this end, we propose a novel multi-modal visual marking approach to annotate the obstacle-free regions in the RGB image used as input to the VLM with numbers, by correlating it with a local occupancy map of the environment. The marked numbers ground image locations in the real-world, direct the VLM's attention solely to navigable locations, and elucidate the spatial relationships between them and terrains depicted in the image to the VLM. Next, we query the VLM to select numbers on the marked image that satisfy the context-based behavior text prompt, and construct a reference path using the selected numbers. Finally, we propose a method to extrapolate the reference trajectory when the robot's environmental context has not changed to prevent unnecessary VLM queries. We use the reference trajectory to guide a motion planner, and demonstrate that it leads to human-like behaviors (e.g. not cutting through a group of people, using crosswalks, etc.) in various real-world indoor and outdoor scenarios.

翻译：我们提出了ConVOI，一种利用视觉语言模型（VLM）在真实室内外环境中实现自主机器人导航的新方法。我们通过两种方式使用VLM：首先，利用其零样本图像分类能力来识别机器人周围环境的上下文或场景（例如室内走廊、室外地形、人行横道等），并将基于上下文的导航行为表述为简单的文本提示（如"保持在人行道上"）。其次，利用其先进的语义理解和逻辑推理能力，在给定已识别上下文的情况下计算合适的轨迹。为此，我们提出了一种新颖的多模态视觉标记方法，通过将RGB图像与环境的局部占据网格图相关联，为输入至VLM的RGB图像中的无障碍区域标注数字。这些标注的数字将图像位置与现实世界对应，将VLM的注意力仅引导至可导航区域，并向VLM阐明图像中这些区域与地形之间的空间关系。接着，我们查询VLM以选择已标注图像中满足基于上下文行为文本提示的数字，并利用所选数字构建参考路径。最后，我们提出一种在机器人环境上下文未发生变化时外推参考轨迹的方法，以避免不必要的VLM查询。我们利用参考轨迹指导运动规划器，并证明该方法在多种真实室内外场景中能产生类人行为（例如不穿人群、使用人行横道等）。