Programming robot behavior in a complex world faces challenges on multiple levels, from dextrous low-level skills to high-level planning and reasoning. Recent pre-trained Large Language Models (LLMs) have shown remarkable reasoning ability in few-shot robotic planning. However, it remains challenging to ground LLMs in multimodal sensory input and continuous action output, while enabling a robot to interact with its environment and acquire novel information as its policies unfold. We develop a robot interaction scenario with a partially observable state, which necessitates a robot to decide on a range of epistemic actions in order to sample sensory information among multiple modalities, before being able to execute the task correctly. An interactive perception framework is therefore proposed with an LLM as its backbone, whose ability is exploited to instruct epistemic actions and to reason over the resulting multimodal sensations (vision, sound, haptics, proprioception), as well as to plan an entire task execution based on the interactively acquired information. Our study demonstrates that LLMs can provide high-level planning and reasoning skills and control interactive robot behavior in a multimodal environment, while multimodal modules with the context of the environmental state help ground the LLMs and extend their processing ability. The project website can be found at \href{https://matcha-model.github.io}{\textcolor{blue}{https://matcha-model.github.io/}}.

翻译：在复杂世界中编程机器人行为面临多层次挑战，从灵巧的低级技能到高级规划与推理。近期预训练的大语言模型在少样本机器人规划中展现出卓越的推理能力。然而，如何将大语言模型锚定于多模态感官输入和连续动作输出，同时使机器人能在策略展开过程中与环境交互并获取新信息，仍是亟待解决的问题。我们开发了一个具有部分可观测状态的机器人交互场景，要求机器人在正确执行任务前，需决定一系列认知行动以从多种模态中采样感官信息。为此，提出以大型语言模型为骨干的交互式感知框架，利用其能力指导认知行动、推理多模态感知结果（视觉、声音、触觉、本体感觉），并基于交互获取的信息规划完整任务执行。研究表明，大型语言模型能提供高级规划与推理能力，控制机器人在多模态环境中的交互行为，而具有环境状态上下文的多模态模块有助于锚定大语言模型并扩展其处理能力。项目网站详见：\href{https://matcha-model.github.io}{\textcolor{blue}{https://matcha-model.github.io/}}。