In order for robots to follow open-ended instructions like "go open the brown cabinet over the sink", they require an understanding of both the scene geometry and the semantics of their environment. Robotic systems often handle these through separate pipelines, sometimes using very different representation spaces, which can be suboptimal when the two objectives conflict. In this work, we present USA-Net, a simple method for constructing a world representation that encodes both the semantics and spatial affordances of a scene in a differentiable map. This allows us to build a gradient-based planner which can navigate to locations in the scene specified using open-ended vocabulary. We use this planner to consistently generate trajectories which are both shorter 5-10% shorter and 10-30% closer to our goal query in CLIP embedding space than paths from comparable grid-based planners which don't leverage gradient information. To our knowledge, this is the first end-to-end differentiable planner optimizes for both semantics and affordance in a single implicit map. Code and visuals are available at our website: https://usa.bolte.cc/

翻译：为使机器人能够执行“打开水槽上方棕色柜子”等开放式指令，它们需要同时理解场景几何结构及其环境语义。机器人系统通常通过独立管线分别处理这两类信息，有时会采用截然不同的表征空间——当两种目标相互冲突时，这种处理方式可能并非最优解。本文提出USA-Net，一种构建统一世界表征的简洁方法，可在可微地图中编码场景的语义信息与空间可供性。基于此，我们构建了基于梯度的规划器，使其能利用开放式词汇导航至场景中指定的位置。实验表明，相较于未利用梯度信息的传统栅格规划器，该规划器生成的轨迹在CLIP嵌入空间中与目标查询的距离缩短了10–30%，且路径长度减少了5–10%。据我们所知，这是首个在单一隐式地图中同时优化语义与可供性的端到端可微规划器。代码与可视化演示详见：https://usa.bolte.cc/