Hierarchical control for robotics has long been plagued by the need to have a well defined interface layer to communicate between high-level task planners and low-level policies. With the advent of LLMs, language has been emerging as a prospective interface layer. However, this has several limitations. Not all tasks can be decomposed into steps that are easily expressible in natural language (e.g. performing a dance routine). Further, it makes end-to-end finetuning on embodied data challenging due to domain shift and catastrophic forgetting. We introduce our method -- Learnable Latent Codes as Bridges (LCB) -- as an alternate architecture to overcome these limitations. \method~uses a learnable latent code to act as a bridge between LLMs and low-level policies. This enables LLMs to flexibly communicate goals in the task plan without being entirely constrained by language limitations. Additionally, it enables end-to-end finetuning without destroying the embedding space of word tokens learned during pre-training. Through experiments on Language Table and Calvin, two common language based benchmarks for embodied agents, we find that \method~outperforms baselines (including those w/ GPT-4V) that leverage pure language as the interface layer on tasks that require reasoning and multi-step behaviors.

翻译：机器人分层控制长期受困于需要明确定义的接口层来连接高层任务规划器与底层策略。随着大语言模型的出现，语言逐渐成为一种潜在的接口层，但这存在若干局限：并非所有任务都能分解为易用自然语言表达的步骤（如执行舞蹈动作），同时由于域偏移和灾难性遗忘，在具身数据上进行端到端微调极具挑战性。我们提出方法——可学习潜码桥接——作为一种替代架构来克服这些局限。该方法利用可学习的潜码作为大语言模型与底层策略之间的桥梁，使大语言模型能灵活传达任务计划中的目标而不完全受限于语言限制，同时可实现端到端微调而不破坏预训练阶段习得的词元嵌入空间。通过在具身智能体常用语言基准Language Table和Calvin上的实验，我们发现本方法在需要推理和多步行为的任务中优于使用纯语言作为接口层的基线方法（包括采用GPT-4V的方法）。