The impressive capabilities of Large Language Models (LLMs) have led to various efforts to enable robots to be controlled through natural language instructions, opening exciting possibilities for human-robot interaction The goal is for the motor-control task to be performed accurately, efficiently and safely while also enjoying the flexibility imparted by LLMs to specify and adjust the task through natural language. In this work, we demonstrate how a careful layering of an LLM in combination with a Model Predictive Control (MPC) formulation allows for accurate and flexible robotic control via natural language while taking into consideration safety constraints. In particular, we rely on the LLM to effectively frame constraints and objective functions as mathematical expressions, which are later used in the motor-control module via MPC. The transparency of the optimization formulation allows for interpretability of the task and enables adjustments through human feedback. We demonstrate the validity of our method through extensive experiments on long-horizon reasoning, contact-rich, and multi-object interaction tasks. Our evaluations show that NARRATE outperforms current existing methods on these benchmarks and effectively transfers to the real world on two different embodiments. Videos, Code and Prompts at narrate-mpc.github.io

翻译：大语言模型（LLMs）的卓越能力催生了多种通过自然语言指令控制机器人的研究，为人机交互开辟了激动人心的可能性。其目标是在确保运动控制任务精确、高效、安全执行的同时，借助LLMs通过自然语言指定和调整任务带来的灵活性。本文证明，通过将LLM与模型预测控制（MPC）框架谨慎分层，可以在考虑安全约束的情况下，实现基于自然语言的精确且灵活的机器人控制。具体而言，我们利用LLM将约束条件和目标函数有效构建为数学表达式，这些表达式随后通过MPC在运动控制模块中使用。优化框架的透明性允许任务可解释，并支持通过人类反馈进行调整。我们通过大量实验验证了该方法在长时序推理、接触密集型及多物体交互任务中的有效性。评估表明，NARRATE在这些基准测试中优于现有方法，并能在两种不同实体平台上成功迁移至真实世界。视频、代码及提示词详见narrate-mpc.github.io。