Recent progress in large language models (LLMs) has demonstrated the ability to learn and leverage Internet-scale knowledge through pre-training with autoregressive models. Unfortunately, applying such models to settings with embodied agents, such as robots, is challenging due to their lack of experience with the physical world, inability to parse non-language observations, and ignorance of rewards or safety constraints that robots may require. On the other hand, language-conditioned robotic policies that learn from interaction data can provide the necessary grounding that allows the agent to be correctly situated in the real world, but such policies are limited by the lack of high-level semantic understanding due to the limited breadth of the interaction data available for training them. Thus, if we want to make use of the semantic knowledge in a language model while still situating it in an embodied setting, we must construct an action sequence that is both likely according to the language model and also realizable according to grounded models of the environment. We frame this as a problem similar to probabilistic filtering: decode a sequence that both has high probability under the language model and high probability under a set of grounded model objectives. We demonstrate this guided decoding strategy is able to solve complex, long-horizon embodiment tasks in a robotic setting by leveraging the knowledge of both models. The project's website can be found at grounded-decoding.github.io.

翻译：大型语言模型（LLMs）的最新进展表明，通过自回归模型的预训练，能够学习并利用互联网规模的语义知识。然而，这类模型应用于具身智能体（如机器人）时面临挑战：它们缺乏对物理世界的体验、无法解析非语言感知输入、且忽视机器人所需的安全约束或奖励机制。另一方面，从交互数据中学习语言条件化机器人策略虽能为智能体提供必要的现实世界锚定，但由于训练数据的广度有限，这些策略的高层语义理解能力受限。因此，若要在利用语言模型语义知识的同时将其锚定于具身环境，必须构建既符合语言模型概率分布又满足环境具身模型约束的动作序列。我们将此问题类比于概率滤波：解码出的序列需同时满足在语言模型下具有高概率，并在系列具身模型目标约束下具有高可实现性。实验证明，这种引导式解码策略能够通过融合两类模型的优势，解决机器人场景中复杂的长时域具身任务。项目网站见 grounded-decoding.github.io。