We present BehAV, a novel approach for autonomous robot navigation in outdoor scenes guided by human instructions and leveraging Vision Language Models (VLMs). Our method interprets human commands using a Large Language Model (LLM) and categorizes the instructions into navigation and behavioral guidelines. Navigation guidelines consist of directional commands (e.g., "move forward until") and associated landmarks (e.g., "the building with blue windows"), while behavioral guidelines encompass regulatory actions (e.g., "stay on") and their corresponding objects (e.g., "pavements"). We use VLMs for their zero-shot scene understanding capabilities to estimate landmark locations from RGB images for robot navigation. Further, we introduce a novel scene representation that utilizes VLMs to ground behavioral rules into a behavioral cost map. This cost map encodes the presence of behavioral objects within the scene and assigns costs based on their regulatory actions. The behavioral cost map is integrated with a LiDAR-based occupancy map for navigation. To navigate outdoor scenes while adhering to the instructed behaviors, we present an unconstrained Model Predictive Control (MPC)-based planner that prioritizes both reaching landmarks and following behavioral guidelines. We evaluate the performance of BehAV on a quadruped robot across diverse real-world scenarios, demonstrating a 22.49% improvement in alignment with human-teleoperated actions, as measured by Frechet distance, and achieving a 40% higher navigation success rate compared to state-of-the-art methods.

翻译：本文提出BehAV，一种利用视觉语言模型（VLMs）并遵循人类指令的户外场景机器人自主导航新方法。我们的方法使用大语言模型（LLM）解析人类指令，并将其分类为导航指南与行为指南。导航指南包含方向性指令（例如“向前移动直至”）及其关联的地标（例如“有蓝色窗户的建筑”），而行为指南则包含约束性动作（例如“保持在”）及其对应的对象（例如“人行道”）。我们利用VLM的零样本场景理解能力，从RGB图像中估计地标位置以支持机器人导航。此外，我们引入一种新颖的场景表示方法，利用VLM将行为规则嵌入到行为代价地图中。该代价地图编码场景中行为对象的存在，并根据其约束性动作分配代价。行为代价地图与基于LiDAR的占据地图集成用于导航。为了在遵循指令行为的同时导航户外场景，我们提出了一种基于无约束模型预测控制（MPC）的规划器，该规划器同时优先考虑抵达地标与遵循行为指南。我们在四足机器人上于多样真实场景中评估BehAV的性能，结果显示其与人类遥操作动作的一致性（以弗雷歇距离衡量）提升了22.49%，且相较于最先进方法实现了40%更高的导航成功率。