Large Language Models (LLMs) pre-trained on internet-scale datasets have shown impressive capabilities in code understanding, synthesis, and general purpose question-and-answering. Key to their performance is the substantial prior knowledge acquired during training and their ability to reason over extended sequences of symbols, often presented in natural language. In this work, we aim to harness the extensive long-term reasoning, natural language comprehension, and the available prior knowledge of LLMs for increased resilience and adaptation in autonomous mobile robots. We introduce REAL, an approach for REsilience and Adaptation using LLMs. REAL provides a strategy to employ LLMs as a part of the mission planning and control framework of an autonomous robot. The LLM employed by REAL provides (i) a source of prior knowledge to increase resilience for challenging scenarios that the system had not been explicitly designed for; (ii) a way to interpret natural-language and other log/diagnostic information available in the autonomy stack, for mission planning; (iii) a way to adapt the control inputs using minimal user-provided prior knowledge about the dynamics/kinematics of the robot. We integrate REAL in the autonomy stack of a real multirotor, querying onboard an offboard LLM at 0.1-1.0 Hz as part the robot's mission planning and control feedback loops. We demonstrate in real-world experiments the ability of the LLM to reduce the position tracking errors of a multirotor under the presence of (i) errors in the parameters of the controller and (ii) unmodeled dynamics. We also show (iii) decision making to avoid potentially dangerous scenarios (e.g., robot oscillates) that had not been explicitly accounted for in the initial prompt design.

翻译：大规模语言模型（LLMs）经过互联网规模数据集的预训练，在代码理解、代码生成及通用问答任务中展现出卓越能力。其核心性能优势源于训练过程中获取的丰富先验知识，以及对以自然语言呈现的扩展符号序列进行推理的能力。本研究旨在利用LLMs的长期推理、自然语言理解及先验知识，增强自主移动机器人的弹性和自适应能力。我们提出REAL（基于LLMs的弹性与自适应方法），该策略将LLMs集成至自主机器人的任务规划与控制框架中。REAL所采用的LLM提供了：（i）先验知识源，用于增强系统未明确设计场景下的弹性；（ii）解读自主堆栈中的自然语言及日志/诊断信息的能力，以支持任务规划；（iii）利用用户提供的少量机器人动力学/运动学先验知识实现控制输入自适应的方法。我们将REAL集成至实际多旋翼飞行器的自主堆栈中，以0.1-1.0Hz的频率查询机载或离机LLM，作为机器人任务规划与控制反馈环路的一部分。通过真实实验，我们验证了LLM在以下场景中降低多旋翼位置跟踪误差的能力：（i）控制器参数存在误差；（ii）存在未建模动力学。同时，（iii）展示了在初始提示设计中未明确考虑的潜在危险场景（如机器人振荡）下的决策规避能力。